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ABSTRACT 


DEVELOPMENT OF RISK UNCERTAINTY FACTORS FROM HISTORICAL NASA 

PROJECTS 

Tahani R. Amer 
Old Dominion University, 2011 
Director: Dr. C. Ariel Pinto 

NASA is a good investment of federal funds and strives to provide the best value to 
the nation. NASA has consistently budgeted to unrealistic cost estimates, which are 
evident in the cost growth in many of its programs. In this investigation, NASA has been 
using available uncertainty factors from the Aerospace Corporation, Air Force, and Booz 
Allen Hamilton to develop projects’ risk posture. NASA has no insight into the 
developmental of these factors and, as demonstrated here, this can lead to unrealistic risks 
in many NASA Programs and projects (P/p). The primary contribution of this project is 
the development of NASA missions’ uncertainty factors, from actual historical NASA 
projects, to aid cost-estimating as well as for independent reviews which provide NASA 
senior management with infonnation and analysis to determine the appropriate decision 
regarding P/p. In general terms, this research project advances programmatic analysis for 
NASA projects. 
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CHAPTER 1 
INTRODUCTION 

“All things are difficult before they are easy. ” 
- Thomas Fuller 


1.1 Background 

NASA’s Space Flight Programs and projects (P/p) are considered highly visible 
national assets and priorities. The Agency’s strategic plan articulates these space flight 
goals and the timetable for reaching them. P/p management translates the strategy into the 
actions needed to achieve these goals. Thus, NASA defines the requirements for effective 
P/p management to fulfill its mandate and commitments. From NASA’s perspective, 
there is a distinction between the Program and project. Program is a strategic investment 
that has a defined architecture, technical approach, requirements, funding level, and 
management structure that initiate and direct one or more projects. A Project is a specific 
investment identified in a program plan and has defined requirements, a life-cycle cost, a 
beginning, and an end. A project yields new or revised products that directly address 
NASA’s strategic needs. 

The purpose of the independent life-cycle reviews (ILCR) of P/p is to ensure mission 
success. These formal reviews, with selected team members, provide an independent 
assessment of emerging designs against plans, processes, and requirements to ensure an 
objective assessment of the design and development plans. By having independent 
experts conduct these reviews, the review team provides a unique view that a P/p may 
have overlooked as a consequence of their close involvement with the ongoing P/p work. 
A major P/p goes through an ILCR, which is the analysis of a proposed P/p by an 
independent team composed of management, technical, and programmatic experts from 
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outside the P/p management authority. It provides NASA management with an 
independent assessment of the readiness of the P/p to proceed. There are three objectives 
for conducting ILCRs: 

1. The Agency wants the P/p to receive independent assurance that they will achieve 
mission success. 

2. The NASA senior management, associate administrators, center directors, and the 
NASA Chief Engineer all need to understand that the P/p is meeting its 
commitments, is perfonning according to plan, and that externally impediments are 
addressed. By conducting ILCR, senior management gains understanding of the P/p 
status and can make informative decisions relative to the P/p. 

3. NASA needs to provide its external stakeholders, such as the Office of Management 
& Budget (OMB), Congress, and policy makers, the assurance that NASA is 
meeting its commitment. Its external stakeholders require reviews at major 
milestones to ensure sufficient management involvement in the decision process 
prior to continuing into the next phase. The intent of ILCRs imposed on P/p is to 
ensure mission success. The Standing Review Board (SRB) is an advisory body and 
can provide recommendations during the key decision points (KDPs) within the P/p 
life-cycle. 

The NASA Convening Authority (CA), which is composed of associate 
administrators, center directors, and the NASA Chief Engineer, reviews these 
recommendations and makes one of the following decisions based on the results of the 
ILCR: 


a. Confirm the P/p to the next phase-continue 
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b. De-scope P/p requirements and objectives 

c. Cancel the P/p 

d. Provide more resources to the P/p to meet requirements 

1.2 Details of the Independent Life-Cycle Review 

A significant additional benefit to the P/p is that preparation for the milestone review 
requires the P/p managers and team to examine holistic progress against specific criteria 
for each milestone. This permits both the development team and the independent review 
team to see how well the work is progressing and to examine the assumptions and 
analyses that support the conclusion the P/p has reached regarding its maturity and 
readiness to proceed. 

The depth of the independent review is to the extent at which the review board can 
determine that the entire design holds together adequately, and that the analyses, 
development work, systems engineering and programmatic (e.g., cost, schedule, etc.) 
support the design and the decisions that were made. Typically, this requires evaluation 
of the work at the system level. Additionally, the independent review function is 
identifying cost, schedule, and technical perfonnance risks as well as identifies the 
consequences of P/p success. 

The independent P/p reviews usually examine the following six criteria for P/p: 

1 . Alignment with NASA Goals 

2. Management Adequacy 

3. Technical Adequacy 

4. Integrated Cost and Schedule Adequacy 

5. Resource Adequacy 
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6. Risk Management Adequacy 

As part of the independent review, focus is on the risk assessment of the P/p. Risk is 
the pressures to meet cost, schedule, and technical perfonnance which are the practical 
realities in engineering today’s systems [Haimes, 2004]. Risk is defined, if it occurs, as 
the combination of the probability that a P/p will experience an undesirable event and the 
consequences, impact, or severity of the undesired event. The undesirable event may 
come from technical or programmatic sources (e.g., a cost overrun, schedule slippage, 
safety mishap, health problem, malicious activities, environmental impact failure to 
achieve a needed scientific or technological objective, or success criterion.) The technical 
and programmatic sources are interdependent and interrelated, thus one cannot separate 
theses sources. Managing risk is managing the inherent contention that exists within and 
across all these dimensions. Both the probability and consequences may have associated 
uncertainties. Risk assessment (see Figure 1) is an evaluation of a risk item that 
determines (Haimes, 2004): 

1 . What can go wrong? 

2. How likely is it to occur? 

3. What are the consequences? 

4. What are the uncertainties associated with the likelihood and consequences? 

5. What are the trade-offs? 


6. What are the future impacts? 
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What can go wrong? 


future impart 



tradeoff? 


alternatives? consequences? 


Figure 1. Six Risk Management Questions (Haimes, 2004) 

These six risk management questions were developed by Haimes in 2004 and have 
been used in the field of risk management since then. 

There are several other organizations and agencies that use the methodology of 
independent reviews for their P/p that have a high level of complexity and have a life- 
cycle cost of $500 million and more. Federal agencies such as the Department of Defense 
(DOD), Federal Aviation Administration (FAA), National Institutes of Health (NIH), and 
Department of Energy (DOE) use independent reviews to assess and evaluate their P/p. 

Independent reviews for a P/p with a life-cycle cost over $250 million are required by 
law to report their progress to Congress and the OMB. Three elements that must be 
evaluated during these reviews are: (1) technical issues, (2) cost, and (3) schedule. 
Stakeholders require an evaluation and integration of these elements. Currently, 
cost/schedule analysts conduct a separate technical, cost, and schedule analysis, not an 
integrated method. Moreover, NASA NPR 1000.5 initiated the requirement to perform 
integrated cost and schedule analyses for major P/p at a specific decision point. 
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1.3 Research Objectives 

The objective of this research is to develop uncertainty factors from NASA’s actual 
historical project data to be used to classify risk for future cost estimations. Additionally, 
it supports the independent reviews which inform NASA senior management to make the 
right decision regarding the project’s progress. This research is to provide a tool to assess 
project risks and provide more informative data for stakeholders and decision-makers. 

1.4 Project Research Problem Areas 

This dissertation focused on four core problem areas. Solution approaches were 
developed for each area in the form of analytic methodologies. 

Problem Area 1 

Determine NASA projects from which to gather data as it relates to cost 
growth for science missions. 

Problem Area 2 

Develop a method to evaluate NASA historical projects’ cost by collecting 
coherent dataset. 

Problem Area 3 

Develop NASA uncertainty factors (NUFs) by capturing the trend of growth 
data from the selected science missions and comparing these factors with 
other uncertainty factors. 

Problem Area 4 

Bring together research and uncertainty factors developed in Problem Areas 1 
through 3 into a coherent tool to be used in the quantification of risk for future 
NASA projects. 
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Figure 2 captures the project problem areas in a graphical format that includes the 
data collection method, data analysis, selection of missions, and testing and validation of 
the results. 


1.5 Research Contribution 

This project’s contribution is to develop NASA mission uncertainty factors from 
actual historical NASA projects to support cost estimating and independent reviews. This 
provides NASA senior management with information and analysis to determine the 
appropriate decision regarding P/p at KDPs. These factors are tested and evaluated by 
statistical methods and the lognormal distribution is developed. 



Problem ^ 
Area 4 + 
Problem 
Area 1 




Problem Area 3 


Findings from pioblern 1 
and 2 will lead to development 
of NASA uncertainty factors. 


Problem Area 1 + Problem Area 2 



Determination from Problem Area 1 will lead to a 
development of method of analysis. 


t; 


\ 



Problem Area 1 


Assessment of NASA historic database and selection of 
40-60 projects. 





Figure 2. Research Project and Problem Area Relationship 


CHAPTER 2 


BACKGROUND OF THE STUDY 

“Strive not to be a success, but rather to be of value. ’’ Einstein 


2.1 Literature Review 

This section separates the literature review into sub-problems in order to be able to 
cover related material (see Figure 3). 





Determine NASA 
Projects to Gather 
Data From 


Develop a method to 
Evaluate NASA Project 
Gost and Schedule data 



Identify Cost 
Growth Causes 


Identify Schedule 
Slip Causes 


Understand Cost 
Growth Versus 
Schedule Growth 


Review software 
tools - @Risk, 
PertMaster, MS 
Project 


Determine 

the relation 

ship betwe 

:n cost and 

schedule 

in NASA 

Projects 




Evaluate F acto rs will 
NASA Current 
Mission 



Figure 3. Research Project Problem and its Sub-Problems Literature Review 
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Risk management has been a major focus of NASA’s culture. Risk management is 
defined by Ruckelshaus (1985) as the process by which the Agency decides what action 
to take in the face of risk estimates. Pinto (2006) has stated that risk management can be 
described in terms of two sets of activities: risk assessment and risk mitigation. Risk 
assessment can be summarized by posing the following questions (Kaplan, 1981): What 
can go wrong? What is the likelihood that it could go wrong? What are the 
consequences? After risks have been assessed, the following questions have to be posed 
for risk mitigation: What can be done? What are the tradeoffs? What are the impacts on 
future options? In this report, sustainable management of risk is accomplished by 
describing frameworks for: (1) valuation of avoided risks, and (2) improving outsourced 
information security services. NASA’s risk assessment method is to avoid risk and 
mitigate it as described by Pinto’s perspective. 

Additionally, Schuyler has defined risk analysis as the discipline of helping decision- 
makers choose wisely under conditions of uncertainty. The quality of decision impacts 
cost, schedule, and perfonnance. Most decision problems are about resource allocation: 
where do we put the money, time, and other resources? Decision analysis involves 
concepts borrowed from probability theory, statistics, psychology, finance, operations 
research, and management science. Also, Schuyler stated that decision analysis provides 
the only logical, consistent way to incorporate judgments about risks and uncertainties 
into an analysis. Decision analysts have to do a credible analysis that must have two main 
characteristics: objectivity and precision. In NASA, decisions are made in all levels of the 
project, but understanding risk that is associated with cost is not a clear concept at 
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NASA. Moreover, Arena (2006) stated that risk analysis has three areas: risk assessment, 
risk management, and risk communication, which interconnect and influence each other. 

Cooper (2003) discussed the relationship between stakeholder expectations and 
project risk. NASA, as the sponsor for interplanetary exploration, provides the funding 
and oversight for the development and operation of all missions. However, the ultimate 
determination of the success or failure of any flight project is the responsibility of the 
stakeholders. The author developed the diagram below (Figure 4) to show the 
relationships of external and internal factors, communication, and inter-connectivity/ 
influence of each other. 



Team Members 


Team Members 


Organization 


Organization 


NASA 


The Public 


NASA 


Affected By 


Exert Influence On 


The Public 


Figure 4. Success and/or Failure of a Flight Project (Cooper, 2003) 
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Cooper’s paper addressed the extension of project risk management practices to 
address the management of stakeholder expectations. This concept establishes the criteria 
for a project’s success/failure and motivates stakeholder actions. The project team will 
understand the stakeholders’ concerns and make decisions that consider the potential 
impacts on the stakeholders. Through his paper is in conceptual phase, it includes a 
stakeholder perspective which has the potential to contribute to the overall risk 
management effort of a project. 

More specific to NASA, Connelly (2004) wrote a paper regarding Integrated Risk 
Management within NASA P/p. This paper states that “the integrating risk across people, 
processes, and project requirements/constraints serves to enhance decisions, strengthen 
communication pathways, and reinforces the ability of the project team to identify and 
manage risks across the broad spectrum of project management responsibilities.” Also, 
the author asserts that applying an integrated approach to risk management makes it 
possible to do a better job at balancing safety, cost, schedule, operational performance, 
and other risk elements. Integrated risk management brings project management and 
engineering processes together to help decision-makers make better decisions. Risk 
management is a deliberate activity, involves a systematic process, and covers the entire 
project life-cycle. Thus, integrated risk management is a process that involves the 
understanding of roles and responsibilities of all stakeholders on the project team. 

Additionally, NASA has a Risk Management Procedural Requirement, NPR 8000.4, 
that is designed to identify, analyze and plan, track, and control risk to increase the 
likelihood of achieving P/p goals. It enables the Project Manager to manage the risks of 
the P/p. It is also a standardization process and a tool to assist all NASA P/p to develop a 
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risk management plan. However, it does not look at the independent review aspect of 
risks and it does not address the evaluation method of risk within the project. 

There is a limited amount of research and documentation regarding NASA’s risk 
management P/p. However, this concept continues to evolve and a great deal of focus and 
energy are being spent to successfully integrate the risk management process across the 
P/p life-cycle. Various processes, tools, and techniques, management involvement, and 
stakeholder monitoring, all must perform and function together to achieve mission 
success. Every P/p undergoes an ILCR, which is the analysis of a proposed P/p by an 
independent team composed of management, technical, and resource experts. This team 
evaluates the cost, schedule, and technical performance, and provides an integrated risk 
assessment of the P/p to senior management. Additionally, the team conducts risk 
analysis, which allows decision-makers to get a better understanding of the range of 
possible outcomes of any decision and to identify kn own risk areas from experts in the 
field. Understanding cost and schedule risks are important components of decision- 
making. Decision-makers seek to understand the risks taken for association with the 
Agency’s investment in order to make an appropriate decision. 

Cost growth is a problem experienced by many types of projects in many fields of 
research. The measurement of cost growth has been inconsistent across programs, NASA 
Centers, and Congress. The Government Accountability Office and Congress generally 
consider the baseline to be the first time a mission appears as a budget line item in an 
appropriations bill, which is often before a preliminary design review. The contents of 
NASA estimates differ from each other and may include: 

Phases A and B, some start with Phase C, 
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Launch costs and/or mission operations, and/or 
NASA oversight and internal project management costs. 

These differences make it difficult to develop a clear understanding of trends in cost 
growth. Thus, different studies reach different conclusions, because they examine 
different sets of missions and calculate cost growth based on different criteria. By 
definition, cost growth is a relative measure reflecting comparison of an initial estimate 
of mission costs against costs actually incurred at a later time (National Research 
Council, 2010). This study considers only development costs. 

Cost growth affects the risk of P/p. There is a great deal of literature that addresses 
risk and risk management in engineering research environments. Figure 3 provides a 
quick look at the literature review of this research. 

NASA P/p conducts internal reviews to establish and manage the P/p baseline. P/p are 
required to document in their P/p Plans their approach to conducting P/p internal reviews 
and how they will support the independent life-cycle reviews. ILCRs are conducted by a 
SRB. 

- The SRB has a single chairperson and a NASA Review Manager. 

- The SRB remains intact, with the goal of having the same core 
membership for the duration of the P/p. 

- SRB members must be independent of the P/p and some members must be 
independent of the Center(s) responsible for the P/p. 



14 


In the article entitled, “Building Better Boards, Harvard Business Review” by David 
Nadler (2004), Nadler discussed the difficulty of board building and the length of time it 
requires. Any board should have certain characteristics, such as the right mind set, the 
right role, the right work, the right people, the right agenda, the right infonnation, and the 
right culture. Also, the article mentioned board building contributes not only to 
performance, but also to member satisfaction as an important element. This article 
provided a relevant perspective on cooperative boards that is very similar to independent 
review boards that evaluate NASA P/p. 

In Dillon paper (2003), he addressed the fact that managers of complex engineering 
development projects face a challenge when deciding how to allocate scarce resources to 
minim i/e the risk of project failure. A new model called the Advanced Programmatic 
Risk Analysis and Management (APRAM), describes a decision-support framework for 
the management of the risk of failures of dependent engineering within projects. The 
model aids the decision-maker in making an informed decision on a top level risk and 
determines the optimal allocation of resources. Also, the model provides a proactive 
approach to making risk take-offs under tight resource constraints. The author concludes 
that NASA is challenged within the current government environment, thus it needs better 
risk management and independent review of technical projects. 

NASA has been and continues to work the risk analysis issue. During a Cost-Risk 
Workshop at Langley Research Center (LaRC), Coonce (2008) stated that “the purpose 
of this workshop was to explain why NASA must improve its cost and schedule 
estimating methods, show forthcoming probabilistic estimating and budgeting policy, and 
explain the fundamentals of probabilistic estimating.” The author stated that NASA’s 
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current projects have exceeded their launch dates by 56% and cost estimates by 64%. The 
cost and schedule growth are adversely affecting other projects in the portfolio as well as 
damaging reputation and credibility with stakeholders. In this workshop, the author stated 
that major NASA projects must submit budgets at a 70% confidence level (CL) starting at 
the initial phase of the project. Figure 5 shows the different cost estimating methods used 
NASA-wide. There are three methods of cost estimating parametric, analogous, and 
engineering (bottom up), for each phase of flight project. Estimates created using a 
parametric approach are based on historical data and mathematical expressions relating 
cost as the dependent variable to selected, independent, cost-driving variables through 
regression analysis. Analogy estimates are performed on the basis of comparison and 
extrapolation to like items or efforts. Cost data from a past program that is technically 
representative of the program to be estimated serves as the basis of the estimate. 

The engineering method is sometimes referred to as "grass roots" or "bottom-up" 
estimating. The engineering build up methodology is rolls up individual estimates for 
each element into the overall estimate. This costing methodology involves the 
computation of the cost of a WBS element by estimating at the lowest level of detail 
(often referred to as the "work package" level), wherein the resources to accomplish the 
work effort are readily distinguishable and discernible. Currently at NASA, there is more 
emphasis on the parametric method. 
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Figure 5. NASA-Wide Cost Estimating Methods (NASA Cost Estimating Handbook, 

2008) 


To realistically implement the 70% CL estimate policy, the P/p must: be completely 
transparent on how their estimate was derived and allow sufficient time for the other 
party to understand it; provide a basis for their respective base estimates; and provide 
rationale and data to explain how they derived their probability distributions. NASA 
Policy Directive (NPD 1000.5) has placed a new requirement on the P/p that P/p must 
comply with the new requirement in order to approve funding. 

There are several developmental processes and methods to integrate the cost and 
schedule that are underway in the risk estimating field. Smart (2007) perfonned research 
on cost and schedule relationships and developed a cost model that implemented funding 
profiles with cost caps, cost impacts on schedule, and schedule impacts on cost. Smart 
stated that cost and schedule are highly correlated. For example, if the schedule slips, the 
cost will increase. Cost and schedule are mathematically correlated, but there is no tested 
and verified model that is equipped to handle cost and schedule jointly. In reality, cost 
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and schedule estimates are analyzed and developed independently of one another. Most 
of NASA P/p incur schedule overruns, thus when schedule increases, costs increase due 
to a stretching of the funding profile. See Figure 6. 



Figure 6. Cost Penalties Due to Changes in Schedule (Smart, 2007) 


In conclusion, Smart stated that cost growth is sensitive to schedule growth and 
developed several algorithms for the effect of schedule expansion, schedule compression, 
and funding caps on cost. His research resulted in NASA beginning an integrated 
approach to cost, schedule, and risk assessment. Moreover, the Quantitative Techniques 
Incorporating Phasing and Schedule (QTIPS) model has been developed from Smart’s 
research and several NASA cost and schedule analysts use this model in their analyses. 

Another method was developed by David Hulett (2007). Hulett presented his paper at 
the 2007 NASA Project Management Challenge. He stated that schedule risk analysis is 
dependent on one-path schedule that has two branches: risk and probabilistic. Schedule is 
managed using Microsoft® Project, but cost is managed using Microsoft® Excel. Hulett 
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developed the pictorial shown in Figure 7 to show integration of cost and schedule on 
project risk. 


Cost and Schedule Risk Integration 
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© 2007 Hulett & Associates, LLC 

Figure 7. Cost and Schedule Risk Integration (Hulett, 2007) 

Additionally, Hulett (2007) stated that schedule risk depends on the schedule logic 
and an uncertainty in the activity duration and also that Monte Carlo simulation is the 
acceptable method of estimating uncertainty from all risks. Cost risk depends on 
schedule uncertainty, uncertainty in burning rates, and uncertainty for time-independent 
costs. 

Moreover, Parsons (2007) stated that problems are better prevented than solved. Data 
is critical for detecting and predicting potential problems; and the purpose of an 
independent review is to predict and plan for any risk that the project cannot detect. The 
independent cost and schedule analysts usually use technical and programmatic data from 
early missions and projects to populate their models. Thus, using a data mining package 
and models to predict future project risk is the core of the independent review’s objective. 





19 


NASA has implemented independent reviews to assess future projects using formal 
project data. 

Steven Grey’s book entitled, “Practical Risk Assessment for Project Management,” 
showed how to accomplish a quantitative cost and schedule risk analysis of projects and 
explained how to apply the same methods to forecasting revenue/pro fits in a project’s 
business. These assessments are conducted independently and are not integrated. 
Additionally, he stated that risk models are evaluated by Monte Carlo simulation, such as 
the @RISK Simulation tool. He addressed the cost risk by assessing the uncertainty in the 
project’s costs, breaking down the total cost into parts, describing the uncertainty in each 
part, and then putting the parts back together to give a whole picture. The standard way to 
break down a project is by the implementation of a work breakdown structure (WBS). 
The schedule risk is represented in terms of a network of li nk ed activities with a logical 
structure, a more complex structure rather than a list of costs to be added. Thus, a basic 
form of a schedule risk model is: a network with all the dependencies between activities; 
a three-point estimate for the durations of all activities including contingencies and lags 
on links; definitions of correlation between estimates; and the probabilities associated 
with branching points. Finally, the author referenced several application tools, such as, 
@RISK for Microsoft® Project, Crystal Ball, Predict, and Monte Carlo by Primavera to 
be used to develop risk assessment of projects. 

2.2 NASA Specifics 

In the 2001 Government Accountability Office (GAO) report, the GAO stated that 
NASA does not have a performance measure that directly addresses the space station cost 
control or risk mitigation activities and contingency planning. The Program lacks a risk 
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management plan and lacks an understanding of all aspects of the risk and its associated 
cost. This report has emphasized the need to understand risk and how it relates to P/p 
success. 

Bitten et al. (2005) have shown that schedule restrictions imposed on planetary 
missions by fixed launch dates create higher failure rates and appear to have more cost 
growth due to schedule restrictions. NASA studies observed that planetary missions fail 
at a rate markedly higher than that of Earth-orbiting missions. They examined the 
relationship between schedule and risk for planetary missions; the data included 38 
NASA missions. They focused on the development time and operational status and found 
that of the 3.9% of missions that experienced schedule growth, 30% were successful, 
40% were impaired, and 30% experienced catastrophic failure rates. They recommended 
that development time for planetary science should be greater than 36 months and should 
be closer to 46 months to be consistent with the average development time for successful 
missions. This research provides a great approach for data analysis of historic NASA 
planetary science missions that could be evaluated for this current research. Additionally, 
it emphasizes the need to understand the cost and schedule relationship. 

Kellogg and Phan (2002) developed an approach for estimating the costs of space- 
based instruments by using actual costs from historical instruments. They tested their 
approach with the NASA Goddard cost model for verification. They concluded that 
analogy based estimating was a powerful tool for cost estimators to use, especially in the 
early conceptual design phase. For this research, uncertainty factors are to be developed 
from NASA historical data, which is similar in methodology to that which Kellogg and 
Phan have recommended. Bitten, Emmons, and Freaner (2005) have addressed the 
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question of funding profile on cost and schedule growth. The initial funding profile 
provided by a mission is one of many factors that can contribute to the cost and schedule 
growth of a mission. The results of their study indicated that certain initial funding 
profiles may minimize cost and schedule growth. Finally, they stated that the best choice 
of funding profile is made after fully understanding the development challenges of the 
mission, the mission development time required to successfully implementing the 
mission, mission requirements, and the mission acquisition approach. The authors have 
provided guidance as follows: 

• A more balanced profile (45%-55% beta curve) may limit cost & schedule 
growth. 

• A more back-loaded funding profile is better for missions with longer 
development times. 

• A front-loaded profile could be managed to retain large reserves during 
early phases that could be carried over to later phases. This option is the 
best, if managed properly, and provides the most flexibility for early risk 
mitigation and responds to problems that occur in integration and testing 
(I&T). 

This study provided correlation between the funding profile, cost, and schedule 
growth, which is an element that needs to be considered within this current project. Also, 
this study provides a primary source of information on NASA’s fiscal year budget. 

Bitten, Emmons, and Freaner have studied NASA cost and schedule growth to set 
reserved guidelines for future P/p. They stated that the current average cost reserve is on 
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the order of 19% and 8% for schedule reserve for each project. From the study of 40 
missions, they recommended an addition of 14% cost reserve at the program level over 
and above the 19% cost reserves that typically has been held at the project level. They 
also recommended increasing the schedule reserve to 19% in lieu of 8%. Additionally, 
they provided best practices for controlling cost and schedule growth in their paper and 
provided a comparison to industry guidelines and rule of thumb. This paper’s approach is 
very clear and relevant to the current project of defining and categorizing the causes of 
cost and schedule growth for 40 missions. NASA did not embrace the result of this paper, 
but NASA has set a new policy since then. 

The National Research Council report of 2010 entitled, “Controlling Cost Growth of 
NASA Earth and Space Science Missions,” has focused on changes in NASA policy that 
would reduce or eliminate the cost growth. The report showed a very interesting trend of 
cost growth in the last several decades (Table 1). 


Table 1. Decadal Trends in Cost Growth for NASA Missions 



Cost Growth 



Average (%) 

Median (%) 

1970s 

43 

26 

1980s 

61, 81 

50, 60 

1990s 

36 

26 


Source: Based on data from Schaffer, 2004 


The major categories for cost growth that were cited in the report are: 

• Overly optimistic and unrealistic initial cost estimates, 

• Project instability and funding issues, 

• Problems with development of instruments and other spacecraft technology, and 


Launch service issues. 
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Additionally, the report correlated data from fourteen NASA missions and developed 
a relationship between cost and schedule growth that is described by the following 
equation: 

y=1.23x +0.13 R 2 = 0.63 

y is predicted schedule growth, x is the expected cost growth predicted, and R 2 is the 
coefficient of determination, which is the proportion of variability in a data set that is 
accounted for by a statistical model. This is a good initial correlation that could be used 
for the future project and its accuracy. 

Furthermore, Bruno & et al. reported the following from a 2006 study: 

• Cost history data for 2 1 of the 24 projects studied shows cost growth. 

• Total growth from Phase B start to Estimate-to-Complete (ETC) at launch for all 
projects studied represents a combined impact of $2 billion to the Science Mission 
Directorate’s (SMD) mission portfolio. 

• Schedule history data indicates schedule slips for 19 of the 24 projects studied. 

• 15 of the projects show a substantially increased rate of internal cost growth after 
Critical Design Review (CDR). 

• Correlations between cost performance and development reserves, cost 
performance and Phase B spending, or cost performance and the percent of funds 
spent up to the CDR could not be found. 

• Although adequate Phase B funding is a necessary condition for project success, it 
is not sufficient to ensure good overall cost performance. 
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These results are very similar to other early NASA studies and it confirmed that NASA 
needs to start looking at the problem from a different perspective. The report provided 
three significant recommendations: 

(1) SMD should provide a stable external environment of fixed requirements, 
funding, and launch services; 

(2) should require projects to improve the quality of early baseline cost and 
schedule estimates, to include a complete and explainable basis of the estimates (BOE) 
with corresponding cost and schedule detail, and include a level of reserves, determined 
by the projects that is commensurate with the implementation risk; and 

(3) should consider minimizing or eliminating blanket reserve level requirements. 

Furthermore, Butts and Linton (2009) have compiled a historical evaluation of cost 

and schedule estimating performance and introduced the Joint Confident Level- 
Probabilistic Calculator (JCL-PC). They claimed the JCL-PC corrects the overly 
optimistic cost and schedule estimates and effectively compensates for the unidentified 
risk events. They also referenced ninety-six historical projects that have an average cost 
growth of 93%, and a median growth of 51%. Finally, they provided nine 
recommendations: 1) include all risks in the JCL analysis; 2) mandate precise criteria for 
the JCL; 3) require all estimates to be created by a bonafide group, like the SRB; 4) 
recognize that cost control is important; 5) require managers to identify all elements that 
cause funding distress; 6) require cost estimate to be submitted in future year dollars; 7) 
require a more specific developmental stage of program; 8) disenfranchise the risk reward 
system; and 9) remove the prevailing stigma that under-runs are unacceptable. 
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Additionally, they have compiled 188 projects’ cost and schedule growth dataset, see 
Appendix H. 

NASA is not alone in a government that has program cost growth. The DOD’s major 
space acquisitions increased approximately $12.2 billion, with 44% from fiscal year 2006 
through fiscal year 2011. The GAO stated that the DOD needs to take more action to 
address unrealistic initial cost estimates of space systems (GAO-07-96). Moreover, in the 
Navy Shipbuilding programs, the Defense Contract Audit Agency (DCAA) criticized the 
shipbuilder’s estimating system, specifically for material and subcontract cost. 

The RAND’s Report (2006) stated, in light of cost growth, DOD senior leaders in the 
Air Force want to generate better cost estimates that provide decision-makers with a 
better sense of the risk involved in the cost estimates they receive. The Air Force Cost 
Analysis Agency and the Air Force cost analysis community want to formulate and 
implement a cost uncertainty analysis policy. The report defined that cost uncertainty 
analysis is an important aspect of cost estimating and benefits decision-making. It helps 
decision-makers understand not only the potential funding exposure, but also the nature 
of risks for a particular program. The report emphasized the cost estimating methods; 
such as Monte Carlo, expert judgment, historical analysis, and sensitivity analysis. 
Finally, the report provided recommendations for a cost risk analysis policy for the DOD 
programs. This report is relevant to the current study because it provides a complete 
summary of cost estimating methods that are used in the DOD and could be used to 
mitigate NASA’s similar causes of cost growth. Additionally, the cost estimating policy 
that the report provided could be implemented at NASA in some versions. Finally, this 
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report confirmed that cost growth of programs is not NASA’s unique problem, but that 
DOD has similar issues and concerns. 

2.3 Current Practice 

In the 2008 NASA Cost Estimating Handbook (CEH), the Cost Risk chapter states 
that NASA is embracing cost risk assessment to improve its reputation with external 
stakeholders to deliver projects on time and within budget. NASA management believes 
that all projects should submit budgets that are based on a quantification of all the risks 
that could cause the project to take longer or cost more than initially anticipated. 
Additionally, NASA has updated its policy to do a better job estimating project cost and 
Program Managers must request budget amounts that reflect a 70% probability that the 
project will be completed at or below this amount. NASA management recognizes it will 
take time to fully implement this policy and has created an interim approach for the FY 
2009 guidance. Moreover, NASA has acted on the findings of the 2004 GAO Report and 
the Space Systems Development Growth Analysis report. The NASA cost estimating 
community is resolved to forecast cost more accurately and to account for risk. Appendix 
B contains the NASA Cost Risk Policy as excerpted from the CEH. The CEH reviews 
new measures NASA is implementing to strengthen its attention to cost risk, including: 

• Distinguishing between uncertainty (lack of knowledge or decisions regarding 
program definition or content) and risk (the probability of a predicted event 
occurring and its likely effect or impact on the program). 

• Identifying the level of uncertainty inherent in the estimate by conducting a cost 
risk assessment. 


Pushing for greater front-end definition to minimize uncertainty. 
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• Resisting the urge to hide or carry uncertainty forward under cost estimating 
assumptions. 

Moreover, NASA must be able “to deliver its P/p on time and within the estimated 
budgeted resources,” as stated by Michael Griffin, the fonner NASA administrator. To 
accomplish this objective, the NASA Administrator, through a series of Strategic 
Management Council meetings, decided that all projects should be budgeted at a 70% CL 
based on the independent cost estimate (ICE), which can be funded by either the project, 
Mission Directorate, or performed by NASA's IP AO. This is one of the most important 
ways that NASA can improve the quality of its cost estimates and, hence, its reputation 
with its external stakeholders (see Appendix B). Additionally, NASA has twelve tenets of 
cost risk (Appendix D) that are developed based on the project risk probability 
distributions. 

As seen from the above reviews, NASA must meet both stakeholder expectations and 
its own policy. Better cost estimating will enhance these expectations and allow the 
Program Manager, Project Manager, and the projects to better communicate the 
program’s cost need. Cost estimates predict future programs’ cost and there is uncertainty 
associated with them. 

Thus, uncertainty analysis should be performed to capture the program risks. NASA 
has been using available uncertainty factors from Aerospace, Air Force, and Booz Allen 
Hamilton (BAH) to develop projects’ risk posture (Appendix F). NASA has no insight 
into the development of these factors, which can lead to unrealistic risks in many NASA 
projects. 
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From the literature, there is not a clear method of addressing the NUFs from historical 
data to assess risk of project. Thus, the development of NASA-specific uncertainty 
factors will provide a better cost estimate to the new P/p and move this field forward to a 
more realistic cost prediction. 
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CHAPTER 3 

RESEARCH METHODOLOGY 

“Well begun is half done,” Aristotle 

The purpose of this chapter is to describe the basic knowledge required to collect and 
analyze cost data. This chapter will cover several areas of data collection methodology, 
data synthesizing, and data analysis. This project used programmatic methodology to 
address its process, which includes collecting data from different sources, evaluating by 
qualitatively and quantitatively logical processes and then developing NUFs, which can 
be generalized to future NASA projects. 

3.1 Data Collection 

The question of cost data availability and relevance merits requires more discussion. 
Most methods of assessing cost risk require some historical data, at levels of aggregation 
that vary widely across the different methods. To set the context regarding the magnitude 
of cost growth and using cost growth as a proxy for cost risk, the NASA historical 
experience of cost growth on fifty missions will be explored. This study of cost growth is 
difficult because of a method for recording project cost, technical issues, and schedule 
data must be developed and implemented. These data are not recorded in a standardized 
format and collected at a reasonable frequency. The depth at which the data are collected 
is not dependent on the maturity of the project. The data is not consistent across the life 
of the project so that, at project end, analysts can evaluate the data across the years 
without ambiguity. 

The goal of this project is to use historical NASA cost growth to develop NUF in 
estimating risk during projects’ initial phases of development. NASA has a vast of 
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sources that house cost information. Over the years, NASA has developed a database to 
document the cost of its missions. Using these data, with other supplementary 
information, this project examined cost growth history to understand the cost growth data 
distribution and to develop specific NASA uncertainty factors. This project has acquired 
the data from three different sources: 

a. NASA Fiscal Year Budget Estimates: 

One source of information for the basis for cost growth is the NASA Fiscal Year 
Budget Estimates. These documents are publicly released in February of each year and 
display the cost and major milestones of NASA’s major programs. Other researchers 
have acquired and collected data on NASA Earth and Space missions to address different 
goals. Bitten et ah, Smart, and Butts’ papers have all investigated recent NASA cost and 
schedule growth history for science missions. These missions included both Space and 
Earth Science missions, Aeronautics, Space Operational missions, and other Programs. 
An examination of this historical data has shown that such space projects often 
experience higher costs relative to initial estimates and project plans. For this study, 
Freaner’s data was investigated and categorized to develop the NASA uncertainty factors. 
Thus, this project used data for forty NASA missions as the basis for the cost growth that 
was collected by Freaner’s team. These missions are shown in Table 2. 

b. Cost Analysis Data Requirement (CADRe): 

It has been difficult to obtain technical and cost information on NASA space flight 

systems. Once a mission was launched, personnel were reassigned and development data 
was lost or thrown away. In December of 2003, NASA initiated a document action 
process that would capture technical and cost information regarding NASA missions at 
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various points during the life of the mission. This document was called the CADRe and 
was incorporated into the NPR 7120.5 series NASA Space Flight Program and Project 
Management Requirements. The CADRe data constitutes one of the better ways to track 
cost estimates and schedules for major NASA missions. Over the past several years, 
NASA has collected and organized cost data from project managers, the budget office, 
and mission directorates as a basis for complete project data. Much of the data for this 
project was obtained from the CADRe that NASA has prepared on each of the missions 
studied. For this project, ten other completed missions have been added to the data. Thus, 
this project will investigate fifty completed missions and ten still active projects’ 
missions (see Table 2). 

c. GAO Reports: 

Several science active missions are included in this study, which were obtained from 
GAO reports and cost analysts from NASA. The GAO report of 201 1 has stated that there 
are 21 NASA projects with a combined life-cycle cost that exceeds $68 billion. This 
report has been used to verily some of the active missions’ data used in this investigation. 
Table 2 provides the data used in this project, which are of two types: completed missions 
and active missions. The active missions are considered an estimate of cost growth. 

Figure 8 summarizes the collection procedure from the three sources and 
demonstrates that data was verified several times to ensure accuracy of the result, which 
created a NASA data set to be evaluated for this project. 
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Figure 8. Project Data Collection Procedure 


3.2 Data Management 

The data for this research project has been managed as described in the flowchart 
found in Figure 9. Sixty missions were collected for this project; and thirty nine were 
used to develop the NASA uncertainty factors. Five completed missions were used to test 


the results. 
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Figure 9. Project Flowchart of Data Collected, Used, Tested, and Assess 


3.3 Project Data Analysis 

Table 2 has the summary of the investigated missions for this project. 


Table 2. Summary of Missions Investigated 


Completed Missions 

Active Missions 

NEAR 

MER 

HESSI 

EOS-Aqua 

AIM 

JUNO 

LUNAR PROSPECTOR 

MRO 

GALEX 

EOS-Aura 

DAWN 

AQUARIUS 

GENESIS 

FAST 

SWIFT 

LANDSAT-7 

PHOENIX 

LDCM 

MESSENGER 

SWAS 

GRACE 

TRMM 

GLAST 

NPP 

MARS PATHFINDER 

TRACE 

CLOUDSAT 

TIMED 

KEPLER 

GPM 

STARDUST 

WIRE 

CALIPSO 

GRAVITY PROBE B 

SDO 

MMS 

CONTOUR 

ACE 

DS-1 

THEMIS 

WISE 

JWST 

DEEP 

FUSE 

EO-1 

HETE-II 

NEW HORIZONS 

MSL 

IMPACT MGS 

IMAGE 

SIRTF 

SORCE 

LRO 

RBSP 

MCO/MPL 

MAP 

STEREO 

ICESAT 

OCO 

GRAIL 
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For this investigation, the development cost is defined as the Phases B-D and does not 
include the launch vehicle cost or operational cost. Figure 10 shows the NASA phases of 
the development from the start of Phase B to the end of Phase D. 
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Figure 10. NASA’s Life-Cycle Reviews for Flight Projects (NASA NPR 7120.5) 


Figure 11 displays two important factors: initial/final cost and the percentage of cost 
growth. The percentage of cost growth of the mission dataset is shown in Figure 1 1 by 
the line chart on the secondary y-axis. For comparison purposes, the development initial 
cost is compared with actual cost from the start of Phase B to the end of Phase D. In this 
chart, the mean of the fifty completed missions’ development cost growth is 
approximately 30%. 
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Figure 11. Initial and Final Cost for 50 Completed Projects including Cost Growth 

Percentage 


Additionally, ten activity missions have been studied in this investigation and have 
seen cost growth in the cost estimating already from PDR to CDR, or SIR, as shown in 
Figure 10. There are two other missions, Mars Science Laboratory and James Webb 
Space Telescope, which were considered for the study, but their cost growth is 1 14% and 
240%, respectively. These two missions are very complex and have a greater funding 
profile than most NASA science missions. Several runs were conducted including these 
two missions; however, the purpose of Figure 12 is to share the other missions’ cost 
growths and ensure they are noticeable. 
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Figure 12. Initial and Current Cost for 9 Active Projects 


3.4 NASA Data Analysis 

To be able to understand the collected data, one must analyze the data in different 
ways to get more insight and understanding. One method is to use the historical cost 
growth as a proxy for the cost uncertainty. This method provides not only the average 
cost growth for past estimates, but also variability in that growth risk. Figure 13 displays 
the percentage of cost growth and the number of completed missions in the histogram 
chart. This chart shows that the data distribution is a bi-modal pattern. Most of the 
mission has expected cost growth of 10-30%, and more risky missions can have cost 
growth of more than 100%, such as MSL and JWST. 
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Figure 13. Percentage of Cost Growth 50 Completed Missions 


When adding the ten active missions, the pattern did not change and the bi-modal 
trend was noted. Figure 14 shows sixty investigated missions that have cost growth and 
have a very similar pattern to the fifty missions. Actually, the greater than 100% active 
missions have increased by 3 times as the completed missions. This indicates that the 
active missions have a higher cost growth than completed missions. 
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Figure 14. Percentage of Cost Growth for 60 Missions 

In 2003, McCrillis developed Figure 15 from the DOD’s 142 systems data entitled, 
“Cost Growth of Major Defense Programs.” This distribution is very similar to the NASA 
sixty missions shown in Figure 14. The similarity contributed to the high risk and 
uniqueness of these two agencies. Additionally, it shows that smaller-sized projects tend 
to have much greater uncertainty in the initial estimates and are the source of the largest 
magnitude cost growth. Moreover, larger-sized projects tend to experience more 
reasonable cost growth, although 100% cost growth is not unexpected. 
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Figure 15. DOD Cost Growth Distributions for 142 Systems (McCrillis, 2003) 

Bearden (2000) stated that each bar’s height represents the percentage difference 
between a satellite’s estimated cost and its actual cost in Figure 16. It has the same profile 
as shown in Figure 14 and Figure 15, which is the bi-modal distribution of the cost 
growth from multiple sources. Bearden explained that a cost-percentage comparison that 
makes use of an older model and the updated dollars-per-kilogram relationships are used 



to estimate modern small-satellite costs. 
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Figure 16. Small Satellite Cost Study (Bearden, 2000) 

Moreover, NASA historical data showed that most of the mission’s cost growth lies 
between 10-30% of its initial cost planned. Figure 17 shows the range from -20% up to 
450% cost growth, which is based on what information was included in the data set. For 
this project, further analysis of the cost growth data indicated that the average total cost 
growth for completed missions was 30% during only the development phase of the 
project. 


Ill 
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Figure 17. NASA 60 Historical Missions Cost Growth 


There are a significant number of missions that experienced cost growth. What are the 
main causes of the growth? One can understand that space businesses are complicated, 
challenging, and one-of-kind. The space industry has been building space-based 
instruments and spacecraft for over 40 years. Experience, lessons learned, and realistic 
planning should reduce the cost growth. Most of the cost growth is classified as related to 
instrument technical development challenges, spacecraft technical problems, due to test 
failures, due to overly optimistic heritage assumptions, and due to management problems. 
A snapshot of the causes of these missions’ cost growth is summarized in Table 3. The 
most commonly identified factors, which have been cited by other formal studies are the 
following: 

• Overly optimistic and unrealistic initial cost estimates, 

• Project instability and funding issues, 
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• Technology level readiness, 

• Assessment of heritage, 

• Problems with development of instruments and other spacecraft technology, and 

• Launch service issues. 


Table 3. Cost Growth Reasons for NASA Selected Missions 



% Cost 


Mission 

Growth 

Reason for Cost Growth 

ACE 

-21.4% 

No major programmatic or technical delays 

NEAR 

-16.8% 

No major programmatic or technical delays 

MGS 

-6.8% 

No major programmatic or technical delays 

EOS-Aqua 

-6.6% 

Launch delay & delayed observatory I&T 

TRMM 

-2.8% 

Launch delay due to Japanese H-II launch vehicle problems 

Stardust 

-0.8% 

No major programmatic or technical delays 

Mars 

Pathfinder 

0.0% 

No major programmatic or technical delays 

Lunar 

1.0% 

Launch delay due to Athena launch vehicle and conflict with Cassini 

Prospector 


launch 

New 

Horizons 

1.2% 

Design stability issue 

MCO/MPL 

3.3% 

Severe programmatic cost and schedule pressure lead to failure of 
both spacecraft 

THEMIS 

5.2% 

Problems with main contractor lead to cost growth 

MAP 

6.7% 

Technical problems due to electronic parts and problems with thermal 
blanketing 

IMAGE 

6.7% 

Launch delay 

LRO 

7.6% 

Heritage complexity, thermal environment and launch delay 

EOS-Aura 

8.1% 

Instrument deliveries delayed due to technical problems 

WISE 

8.2% 

Design stability, Structural model failure during vibration testing 

FUSE 

9.4% 

Problems with Fine Guidance Sensor lead to delay 

SORCE 

9.6% 

Delays in bus and instrument deliveries 

GRACE 

11.5% 

Problems with instrument development led to delays 

DS-1 

12.6% 

Technical problems with introducing new technologies 

TRACE 

13.2% 

Difficulties with instrument development 

MRO 

14.1% 

Problems uncovered during environmental testing 

Landsat-7 

16.0% 

Instrument deliveries delayed due to technical problems 
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% Cost 


Mission 

Growth 

Reason for Cost Growth 

MER 

16.1% 

Resolving mass & schedule problems 

Genesis 

18.5% 

Spacecraft subsystem late delivery then launch delay 

FAST 

18.9% 

Launch vehicle delay caused by Pegasus failure 

SDO 

21.6% 

Complexity of heritage technology, contractor performance, and 
funding issues 

OCO 

22.7% 

Design stability and contractor performance 

Contour 

24.3% 

Technical and staffing problems at APL 

TIMED 

25.6% 

Launch slip due to technical problems encountered by the Jason 1 
spacecraft 

DAWN 

25.8% 

Technical problems, launch date, contractor performance, heritage 
complexity 

WIRE 

27.7% 

Difficulties with instrument development 

Kepler 

29.5% 

Substantial problems with instrument development & contractor 
performance 

Deep Impact 

30.0% 

Technical and management problems at Ball 

STEREO 

31.3% 

Problems with instrument growth & staffing 

SIRTF 

36.3% 

Substantial problems with instrument development and software on 
spacecraft 

AIM 

40.8% 

Difficulties with instrument development 

Gravity 
Probe B 

42.4% 

Cost growth due to various unanticipated technical challenges, late 
delivery of payload. 

Phoenix 

43.2% 

Technical Problems 

HESSI 

45.6% 

Test equipment failure and launch delay due to Pegasus 

ICE SAT 

45.9% 

Significant delays in GLAS instrument development 

Messenger 

51.0% 

Late delivery of instruments and integration problems 

GALEX 

57.9% 

Severe problems with telescope and spacecraft 

GLAST 

59.3% 

Design stability and complexity of heritage technology 

SWIFT 

61.0% 

Problems developing BAT instrument 


66.8% 

Launch vehicle delay caused by Pegasus failure - Spacecraft went 

SWAS 


into storage 

HETE-II 

71.4% 

Problems requiring additional testing and delays with Pegasus Launch 
Vehicle 

CLOUDSAT 

79.6% 

Problems with High Voltage Power Supply combined with Launch 
Vehicle Delay 

EO-1 

114.2% 

Technical Problem, adding Hyperion instrument and launch delay 
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Mission 

% Cost 
Growth 

Reason for Cost Growth 

CALIPSO 

149.7% 

Problems with LIDAR instruments combined with Launch Vehicle 
Delay 

MMS 

3.3% 

Technology maturity and partner performance 

GRAIL 

1.2% 

Heritage complexity and launch manifest 

LDCM 

15.4% 

Technology maturity and partner performance 

GLORY 

17.8% 

Technology maturity, contractor performance and design stability 

JUNO 

20.0% 

Design stability and development partner performance 

GPM 

22.3% 

Technology maturity, and funding issues 

AQUARIUS 

25.7% 

Design stability and partner performance 

NPP 

50.0% 

Technology maturity and design stability 

MSL 

114.3% 

Technology maturity and design stability 

JWST 

242.3% 

Complexity of heritage technology and funding issues 



Most of the last Ten missions are in CDR or SIR 


3.5 Non-NASA Uncertainty Factors 

NASA has historically underestimated and underfunded the cost of new missions. In 
the NASA environment, cost data is limited and accurate cost estimating is a significant 
challenge, as stated previously. Given this environment of limited data and substantial 
uncertainty associated with predicting the future, for best decision support, it is 
imperative that analysts quantify the confidence or uncertainty of their estimates. Cost 
analysts try to develop the best cost estimates possible from the available infonnation. 
The most common approach is to develop a “most likely”, “optimistic”, and “pessimistic” 
estimate for each component in the mission. Because every assumption that drives a cost 
estimate represents a point within a range of possible values, an estimate of this type is 
called the “point estimate,” which carries specific risk within. No matter how much effort 
is applied to the lower elements in the estimate, total levels in the point estimate do not 
reflect a “most likely” value in most cases. 
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Uncertainty occurs for a number of reasons. The objective of the cost uncertainty 
analysis is to estimate the uncertainty of the point estimate and provide a basis for 
assessing its uncertainty or variability for a specific estimate. Because the point estimate 
is based on assumptions with associated uncertainty, the analyst must consider risk and 
uncertainty from the very outset of the project or estimate. Uncertainty is sometimes 
expressed as a probability distribution of outcomes; the greater the width of the 
distribution, the more uncertain the outcome. Uncertainty factors play an important role 
in cost estimation because the amount of uncertainty around an estimate is information 
that helps the decision-maker. 

Additionally, risk analysis is an important component of a decision-making process. 
It allows decision-makers to get a better understanding of the range of possible outcomes 
of any decision — in other words, how good or bad the outcome might be and how 
uncertain the outcome is. Risk analysis also helps the decision-making process by 
identifying known risk areas. In some cases, such infonnation can be used to mitigate 
areas that are high risk. Risk analysis brings more information which, in turn, generates 
more realistic expectations. 

Uncertainty in cost estimating creates concern. Uncertainty of an estimate is tied to a 
risk: the more uncertain the estimate, the greater the chance of an adverse or unexpected 
outcome. Uncertainty of an estimate can reflect both financial risk and operational risk to 
the Agency, which can damage its image and reputation. Thus, to characterize cost 
uncertainty is to characterize cost risk. Understanding cost risk is an important 
component of decision-making. Decision-makers seek to understand the risks they 
assume with any type of investment or program. Greater cost risk might require increased 
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management oversight to reduce or mitigate the risks identified, or to provide more 
reserve funds. 

NASA cost analysts are using three different types of uncertainty factors that were 
developed by: 

1 . The United States Air Force, 

2. The Aerospace Corporation, and 

3. Booz Allen Hamilton (BAH). 

NASA has no insight into the methodology and the data that developed these factors; 
and they do not represent NASA missions. Table 4, Table 5, and Table 6 display the Air 
Force, Aerospace Corporation, and BAH uncertainty factors, respectively. The 
Aerospace UFs are obtained from the 2010 Aerospace Study of twenty SMD projects. 
BAH UFs are obtained from a study that BAH perfonned; this information was 
proprietary. These factors have been used for several NASA missions, such as MMS and 
LDCM. Finally, the Air Force has been used in several NASA missions such as Global 
Precipitation Measurement (GPM) and Radiation Belt Storm Probe (RBSP). 



Table 4. Air Force Uncertainty Factors 


Level 

Optimistic 

Most Likely 

Pessimistic 

Low 

0.95 

1 

1.1 

Low Plus 

0.96 

1 

1.23 

Moderate 

0.97 

1 

1.36 

Moderate Plus 

0.98 

1 

1.49 

High 

0.98 

1 

1.61 

High Plus 

0.99 

1 

1.74 

Very High 

1 

1 

1.87 

Very High Plus 

1 

1 

2 


Table 5. Aerospace Corporation Uncertainty Factors 


Level 

Low 1 0% 

Mid 50° o 

High 90° « 

SEE 

Low 

0.977 

I 

1.117 

0.0: 

Medium Low 

0.932 

1 

1.35 

o.i ; 

Medium 

0.88" 

i 

1.583 

0.25 

Medium High 

0.842 

1 

1.816 

0.35 

High 

0.797 

i 

2.05 

0.45 

:ii£ll+ 

0."52 

1 

2.5 


V ery High 

F 

® 

F 

© 

1 

3 


Very high + 

0.662 

1 

4.5 


Extra high 

0.617 

1 

6 
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Table 6. Booz Allen Hamilton (BAH) Uncertainty Factors 



Current Baseline Schedule 

Complexity 

Aggressive 

Most Likely 

Conservative 

Ve ry Low 

N(1.20, .05) 

N(1.00, .05) 

N(0.8, .05) 

Low 

N(1.20, .15) 

Nfl.OO, .15) 

N(0l8. .15) 

Moderate 

N(1.20, .25) 

N(1.00, .25) 

N(0.8, .25) 

High 

N(1.20, .35) 

N(1.00, .35) 

N(0l8, .35) 

Very High 

N(1.20. .45) 

Nfl.Op, .45) 

NfO_fL .45) 



Assumptions 
-Used conservative of 0 8 and added 1 sigma 
-Used pessimistic of 1 2 and added 2 Sigma 


As noted above, this lack of information about the methods used to develop these 
factors introduced skepticism. These factors have been tested in the NASA historical 
completed missions in the middle range only, as shown in the yellow highlighted rows. 
The approach is known as the 3 -point range. This approach is currently being used by 
NASA to report project cost estimates. In fact, factors are called an estimate in terms of 
high, mid, and low points and may in fact be easier for the experts to provide rather than 
identifying a specific value. Figure 18a displays an example of cost risk using the 3-point 
range method. Each point -optimistic, most likely, and pessimistic -represents an 
estimate with a different set of assumptions (see Figure 18b). These assumptions can 
directly reflect the project’s specific technical and programmatic risks. These points are 
the possibility of showing a 3 -point range determined by a probabilistic assessment. 
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Figure 18a. A Prototype of 3-Point Range Estimate 



Figure 18b. Another Diagram to Demonstrate 3-Point Range Estimate 


For the purposes of this project, three risk levels have been selected to test: moderate, 
high, and high plus. Additionally, the data was tested and analyzed for optimistic and 
pessimistic cases only. These cases are shown in Table 7. 


Table 7. Selected Uncertainty Factors for Three Risk Levels 


Level of Risk 

BAH Uncertainty Factors Selected 


Optimistic 

Most Likely 

Pessimistic 

Moderate 

1.050 

1 

1.700 

High 

1.150 

1 

1.900 

Very High 

1.250 

1 

2.100 

Air Force Uncertainty Factors Selected 

Moderate Plus 

0.98 

1 

1.49 

High 

0.98 

1 

1.61 
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High Plus 

0.99 

1 

1.74 

Aerospace Uncertainty Factors Selected 

Medium High 

0.842 

1 

1.816 

High 

0.797 

1 

2.05 

high+ 

0.752 

1 

2.5 


The NASA data collected for cost risk analysis will be used to empirically validate 
three uncertainty factors and their associated level of risk analyses. Such a validation 
would help to improve both the understanding of the given uncertainty values and quality 
of these factors in the estimation of NASA cost risk process. It is vital to the credibility of 
both cost estimates and cost risk analyses to demonstrate how well they have predicted 
NASA mission cost. Figure 19 displays the NASA historical data using the Air Force 
uncertainty factors for optimistic and pessimistic. Using the Air Force optimistic 
uncertainty factors does not provide the correct prediction of NASA missions as seen in 
the figure. Keep in mind that the initial cost is often budget-driven. For example, for the 
AO missions, the initial “cost” is really the “budget” that the project has been given. For 
the optimistic case, these factors missed the actual final NASA cost of almost 90% of the 
missions, but for the pessimistic case, they overestimated all NASA missions. 
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Figure 19. Air Force Optimistic & Pessimistic Uncertainty Factors & NASA 

Historical Missions Data 


Figure 20 displays the NASA history data and applies the Aerospace uncertainty 
factors for optimistic and pessimistic. As seen from the figure, these factors do not come 
close to the actual NASA data. Thus, they are not valid factors to use to predict cost 
estimation of NASA missions. As seen in Figure 20, the optimistic case, it was 
underestimating NASA data and, for the pessimistic case, these factors predicted a much 
higher estimate than the actual NASA cost. 
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Figure 20. Aerospace Optimistic & Pessimistic Uncertainty Factors & NASA 

Historical Missions Data 


Figure 21 displays the NASA history data and applies the BAH optimistic and 
pessimistic uncertainty factors for optimistic and pessimistic. As seen from the figure, 
these factors do not come close to the actual NASA data. Similar observations have been 
noticed for the BAH factors. For the pessimistic case, they overestimated the NASA cost 
growth missions, but the optimistic was close to the actual, except for a few missions. 
The high risk factors do not estimate or predict NASA actual cost estimation. 
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Figure 21. Booz Allen Hamilton Optimistic & Pessimistic Uncertainty Factors & 

NASA Historical Missions Data 


Thus, the above figures demonstrate that the Air Force, Aerospace and BAH 
uncertainty factors are not the best tools to use to estimate or predict NASA missions’ 
cost risks. It seems clear that no one uncertainty factor can predict the NASA missions 
and assess cost risk in projects. To have a useful and credible cost risk analysis, 
uncertainty factors must be used which fit the level of detail required and the resources 
available for NASA projects. Thus, NASA historical data will be of great value in 
developing actual NUFs. 
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3.6 Understanding the NASA Patterns 

Traditionally, cost uncertainty is communicated through probability distributions, the 
results of a Monte Carlo simulation that are presented through a probable density 
function (PDF) or a cumulative distribution function (CDF). These methods provide the 
decision-maker with the probability distribution of the confidence of an estimate. Often, 
decision-makers are not trained or current in probability methods; thus, their 
understanding of the implied cost uncertainty may be limited. 

NASA historical data for the sixty missions were analyzed in different matters to 
understand the pattern of cost growth. Figure 22 shows how most of the projects have 
cost growth in a range from 30-36%. Additionally, the data was sorted by chronological 
order to provide any indication of whether NASA is improving in cost estimating. 



Figure 22. 60 NASA Historical Data including Initial and Actual Cost 
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Further analysis has been conducted to group the missions and to understand the data. 
Figure 23 shows four different binnings of the cost growth of NASA missions. It seems 
that most of the cost growth occurs between 10-30%. Over 50% of the data fall in that 
range, thus placing the data into bins helps the analysis and provides a manageable group 
data. 




Figure 23. NASA Historical Cost Data with Size of Growth Binning 


It has been selected 10% binning of the data to develop a distribution fit to the 
collected cost growth for fifty completed missions and ten active missions. Additionally, 
the collected data that is higher than 70% cost growth is grouped as one for ease of 
analysis (Figure 23). 
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3.7 Development of Distribution 

This section addresses the development of a specific distribution from the NASA 
sixty missions cost growth data. There is a large number of possible distribution shapes 
defined in the literature, which are available through a variety of tools. In an effort to 
ensure the quality of the result, several distributions defined in Table 8 have been tested. 
Additionally, several software tools have been evaluated to conduct this task: Table 
Curve 2D, EasyFit, and Peak Fit. EasyFit was the selected software to analyze NASA 
data due to the fact that it was compatible with Microsoft® Excel, which is the software 
chosen to store the data. Additionally, EasyFit software is striving for a good balance 
between the accuracy and speed of calculations. It uses the Method of Moments (MOM) 
and the Maximum Likelihood Estimation (MLE). Moreover, it is a part of the MathWave 
data analysis and simulation software that has been in use for decades. Schittkowski 
(1998) has developed a paper explaining the EasyFit software system for data fitting in 
dynamic systems. 

Project cost is an uncertain quantity and probability distributions are used. Triangular, 
Beta, Lognonnal, and Normal are probability distributions commonly used in cost 
estimating uncertainty analysis. Figure 24 graphically demonstrates that point estimates 
of individual elements using the triangular and normal distributions can be quantified as 
most-likely, median, mean, and mode. Perlstein, Jarvis, and Muzzuchi (2001) discuss the 
use of the beta distribution for quantifying cost uncertainty. 
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Figure 24. Statistics of the Triangular and Normal Distributions (NASA CEH, 2008) 

It is important to understand that the actual cost of a project is the cumulative effect 
of small influences. When these influences are additive, use of the normal distribution is 
justified by the Central Limit Theorem (CLT). The CLT states that “the average of the 
sum of a large number of independent, identically distributed random variables with 
finite means and variances converges “in distribution” to a normal random variable” (see 
Figure 25). When the influences are multiplicative, use of the lognormal distribution is 
justified. In general, costs tend to accrue in a multiplicative sort of way, for example, 
wage rate multiplied by headcount. In this investigation, the normal, lognormal and other 
distributions commonly used in cost estimating uncertainty analysis were tested as best- 
fits for observed cost growth on many NASA projects. A list of most common probability 
used in cost estimating uncertainty analysis (shown in Table 8 from the GAO 09-3SP 
report) was evaluated. 
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WBS Element Triangular Merge WBS Element Cost Distributions 

Cost Distributions into Total Cost Normal Distribution 


Most Likely 



Figure 25. Central Limit Theorem (NASA CEH, 2008) 
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Table 8. Common Probability Distributions (GAO 09-3SP Report, 2009) 


Distribution 

Description Shape 




Typical application 

Bernoulli 

Assigns probabilities of "p" for 
success and "1 - p* for failure; 
mean - “p"; variance - *1 - p* 

Probability 



With likelihood and consequence 
risk cube models; good for 
representing the probability of a 
risk occurring but not for the impact 
on the program 






0 


Values 

Beta 

Similar to normal distribution 

^robabifity 




Tc capture outcomes biased toward 


but does not allow for negative 





the tail ends of a range; often used 


cost or duration, this continuous 

» / 




with engineering data or analogy 


distribution can be symmetric or 

/ / 




estimates; the shape parameters 


skewed 



Sh 


usually cannot be collected from 





Values 

interviewees 


Lognormal 


A continuous distribution Pratot* 

positively skewed with a 
limitless upper bound and 
known lower bound; skewed to 

the right to reflect the tendency 

toward higher cost 


To characterize uncertainty 
in nonlinear cost estimating 
relationships; it is important to 
know how to scale the standard 

deviation, which is needed for this 

Vl **‘ distribution 


Normal 


Poisson 


Used for outcomes likely to 
occur on either side of the 
average value; symmetric 
and continuous, allowing for 
negative costs and durations. 

In a normal distnbution, about 
68% of the values fall within one 
standard deviation of the mean 


Peaks early and has a long tail 
compared to other distributions 


Prooobtty ^ 



VtfUN 


Probotortffy 

V 


7 





To assess uncertainty with cost 
estimating methods, standard 
deviation or standard error of the 
estimate is used to determine 
dispersion. Since data must be 
symmetrical, it is not as useful 
for defining risk, which is usually 
asymmetrical, but can be useful for 
scaling estimating error 

To predict all kinds of outcomes, like 
the number of software defects or 
test failures 


Triangular 


Uniform 


Characterized by three points 
(most likely, pessimistic and 
optimistic values), can be 
skewed or symmetric and is 
easy to understand because it 
is intuitive; one drawback is the 
absoluteness of the end points, 
although this is not a limitation 
in practice since it is used in a 
simulation 



To express technical uncertainty, 
because it works for any system 
architecture or design; also used to 
determine schedule uncertainty 


Vtfue* 


Has no peaks because all values, 
including highest and lowest 
possible values, are equally likely 


PiobataMy 

. •■=■ . “ •: r 


With engineering data or analogy 
estimates 


Weibull 

Versatile, can take on the 
characteristics of other 

Piototiity 

In life data and reliability analysis 
because it can mimic other 


distributions, based on the value 


distributions and Its objective 


of the shape parameter ‘b’ — 
e g., Rayleigh and exponential 
distributions can be derived 
from it" 


relationship to reliability modeling 

Virtues 
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The most popular distribution shapes tested in this investigation with sixty NASA 
missions’ data include: lognormal, log logistic, Weibull, Normal, Beta, Burr, and general 
extreme value. Figure 26 displays the sixty missions as functions of cost growth 
percentage plotting with PDF and the probability-probability (P-P) plot. The P-P plot is a 
graph of the empirical CDF values plotted against the theoretical CDF values. It is used 
to determine how well a specific distribution fits to the observed data. This plot will be 
approximately linear if the specified theoretical distribution is the correct model. These 
plots are used as visual and qualitative assessments. 



P-P PCI NASA 60 UlltO 



Figure 26. Probability Density Function & Probability-Probability Plot for 60 

NASA Missions 
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Figure 26 shows that Beta, Burr, Log-Logistic and lognormal distributions fit the data 
well. Table 9 displays the summary of the distribution parameters. 


Table 9. 60 Missions Data Fits 


Distribution 

Parameters 

Beta 

0^=0.54859 a 2 =4.1218 
a =-21.0 b =444.0 

Burr (4P) 

k=0 .42648 a=5.87 
P=49.091 y=-42.854 

Gen. Extreme Value 

k=0.3793 1 a=20.12 u=12.455 

[nv. Gaussian (3P) 

X=128.48 |u=66.62 y=-30.636 

Log-Logistic (3P) 

a=2.8108 p =46. 552 y=-26.488 

Lognonnal(3P) 

cr=0. 66093 u=3.9014 y=-28.101 

''formal 

ci=67.06 (a.=3 5 .984 

Pearson 6 (4P) 

ai=878.35 oc 2 =L143 
P=0 .25804 y=-38.756 

Weibull (3P) 

a=l .1225 P=59.975 y=-2 1 . 1 42 


Several tests from the EasyFit software test the quality of the fit of two tests: the 
Kolmogorov-test and the Anderson-Darling test. The Kolmogorov-Smirnov statistic (D) 
is based on the largest vertical difference between the theoretical and the empirical 
cumulative distribution function. The Anderson-Darling procedure is a general test to 
compare the fit of an observed cumulative distribution function to an expected 
cumulative distribution function. This test gives more weight to the tails than the 
Kolmogorov-Smirnov test. Appendix G has the summary for all the cases that have been 
tested for this project. 

Figure 27 shows the cost growth data for only fifty-four NASA missions, which do 
not include under cost missions. Table 10 displays the result of the distribution fits. 
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Figure 27. Probability Density Function & Probability-Probability Plot for 54 

NASA Missions 
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Table 10. Summary for the 54 Mission Quality of the Fit 


Distribution 

Parameters 

Beta 

ai=0.3319 012=2.789 
a=-6.2932E-17 b=5.4851 

3un 

k=l .5284 a=1.3144 (3=0.34407 

Gen. Extreme Value 

k=0 .49034 c>=0. 15481 (a=0.1848 
\ 

[nv. Gaussian 

>.=0.15329 |u=0 .41833 

Log-Logistic 

a=l .4856 (3=0.22092 

Lognonnal 

a=1.238 n=-1.5616 

Normal 

ct= 0.69109 p.=0 .41833 

3 earson 6 

0d=1.4119 0(2=2.4747 (3=0.43966 

Weibull 

a=0. 84335 (3=0.38227 


The data presented in the three tables above lead to narrowing the focus of Weibull 
and lognonnal distributions. Several other runs have been conducted to understand the 
actual behavior of the data, as follows: 

1. Delete two missions that are over 150% cost increase (see Figure 28). They may 
be outlier missions within the data set. Outliers were checked by determining if 
the fit got better by not including any one mission in the data set at a time. 

2. Include only increase cost - Delete under-cost missions and two missions above 
(see Figure 29). 

3. Delete missions over 100% cost growth (see Figure 30). 

4. Concentrate on the two distributions: lognormal and Weibull (see Figure 31). 
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Figure 28. Probability Density Function & Probability-Probability Plot for 58 

NASA Missions 
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Figure 29. Probability Density Function & Probability-Probability Plot for 52 

NASA Missions 
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Figure 30. Probability Density Function & Probability-Probability Plot for NASA 

Missions less 100% Cost Growth 
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Figure 31. Probability Density Function & Probability-Probability Plot for 50 
NASA Missions -Focus on Two Distributions with Less 100% Cost Growth 
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Numerous studies have empirically shown the lognormal and Weibull to be excellent 
approximations to the overall distribution function of a mission’s total cost, even in the 
presence of correlations among cost element costs. The lognonnal is similar to the 
Weibull, but lognormal is different than the Weibull distribution because it is skewed 
towards the positive end of the range and captures more of the initial missions in the 
lower cost growth instead of being flat in the beginning of the distribution. The 
lognormal distribution illustrates the distribution shape if the cost growth bounds are 
taken as “10-30%,” which seems normal for most missions (see Figure 32). The three- 
parameter lognonnal distribution is: 

a- continuous parameter(a>0) 

p- continuous parameter 

y - continuous location parameter (y= 0 yields the two-parameter lognormal distribution) 
Domain y <x< +oo 

• Probability Density Function 

F(X)= exp (-1/2 (In (x- y )- p / a ) 2 )/ (x- y) a (27t) 1/2 

• Cumulative Distribution Function 


F(x) = cp (In (x- y )- p/ a) (p is the Laplace Integral 
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Lognormal: 

0=0.66. g=3.90. y— 28.1 



Figure 32. Probability Density Function & Probability-Probability Plot for 60 
NASA Missions -Focus on Lognormal Distributions 


Galton (1879) stated that the lognormal is used when considering the multiplication 
or the geometric mean for independent random variables, and the CLT applies to their 
logarithms. The lognormal distribution shape is selected distribution that is ultimately 
modeled in the analysis and has the following property: mean of 3.03 and standard 
deviation of 1.08. This distribution is based on the 44 completed missions with cost 
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growth only. Thus, this lognormal distribution has been selected to describe the NASA 
historical data distribution (see Figure 33): 


F(X) = 


E X P ( ~ 1/2(~~~~) A 2) 
xa(2n) A 2 


)Lt= 3.03 


c= 1.08 


The parameters denoted // and a are the mean and standard deviation, respectively, of 
the variable’s natural logarithm. Additionally, those parameters that can be converted to 
nonnal /i and a, are the mean and standard deviation, respectively, then the values are: 

H=20.7 

< 7 = 2.91 
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Figure 33. Probability Density Function for 44 NASA Missions -Focus on 


Lognormal Distribution and its Property 

Several runs have been conducted to test the robustness of the selected distribution. 
Below are four lognormal distributions from three NASA cost data sets: 

1. All data including MSL and JWST (A- Data), 54 missions (Figure 34). 

2. All data excluding MSL and JWST (B- Data), 52 missions (Figure 35). 

3. All completed missions (C-Data), 44 missions (Figure 33). 

4. All data less than 100% cost growth (D- Data). 
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Table 11. Four Lognormal Distributions Properties for NASA Data 


Type of Data 

o= Mean 

% Error 

p =Std. Dev. 

% Error 

A - Data 

1.18 

6 

3.01 

3.08 

B - Data 

1.11 

2.78 

2.92 

3.63 

C - Data 

1.08 

0.9 

3.03 

6.32 

D-Data 

1.07 


2.85 




Figure 34. Probability Density Function for 54 NASA Missions 
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Figure 35. Probability Density Function for 52 NASA Missions 


The lognormal distribution is the most suitable distribution that a cost estimator can 
use to perform an uncertainty analysis for the NASA data (see Appendix H for lognormal 
model). Lognormal distribution has a defined lower bound that is never less than zero and 
an upper bound of infinity, which provides at least some probability of a large cost 
overrun, as seen in several missions. It is sufficient and represents the characteristic of the 
NASA historical data. Dataset C and Figure 33 are the recommended data to represent 
NASA historical data and will be used to develop the NUFs. 
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CHAPTER 4 

RESULTS AND DISCUSSION 

“The more original a discover y, the more obvious it seems afterwards ” 

- Arthur Koestler 

This chapter provides an overview of the development of the NUF and the validation 
of these factors with NASA data. NUF begins with a general understanding of the 
lognonnal property, and then specifically addresses the factors with validation of several 
missions. 

4,1 Development of NASA Uncertainty Factors 

One of the major considerations in cost estimating is how to assess and quantify 
technical, cost, and schedule risks. Certainly, there are many complex methods and 
formulas that do so, but these risks are ultimately subjective and judgmental in nature, no 
matter how they are developed and applied. The intent of this project is to provide a 
means and rationale for estimating mission risk using a common-sense, non-statistical 
approach that generates results using historical data that correlate well with more 
mathematically rigorous methods. 

Uncertainty is expressed in a simulation by specifying the shape and bounds of the 
uncertainty distribution for the cost methods and cost drivers (input variables) where the 
value is not certain. Understanding the NASA data distribution could be used as a 
lognonnal distribution model from the previous chapter. Lognormal distributions have a 
defined lower bound that is never less than zero. Lognormal does not permit a negative 
tail and preserves the mean and standard deviation. They have an upper bound of infinity, 
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thus providing at least some probability of a large cost overrun. The skew of a lognormal 
is pre-defined. 

The development of the uncertainty factors from NASA data will now be developed 
and demonstrated. Graphing the lognormal distribution with NASA data in a Q-Q plot, 
85% of data has shown falls in a normalized line in the 0-30% range. Figure 36 shows the 
NASA data is mostly below 40% cost growth. Additionally, there are four data points 
that cover missions with cost growth over 100% that do not lie within the Q-Q plot. 
These missions are outside the normal NASA historical cost growth and could be 
skewing the plot, but they needed to be included. The statistical analysis of the historical 
data to derive the growth uncertainty factor is straightforward. It has the additional 
advantage that it takes into account the trends in costs that are not explicitly used in the 
historical analogy methodology. 
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Figure 36. Q-Q Plot of the 54 NASA Missions - Focus on Lognormal Distribution 


If one concentrates on the completed missions mentioned above, then focus on the 
actual repeated cost growth can be recommended for more realistic uncertainty factors. 
Figure 37 shows the Q-Q plot for a mission that has cost growth less than 100%. The 
quantile-quantile (Q-Q) plot is a graph of the input (observed) data values plotted against 
the theoretical (fitted) distribution quantiles. Both axes of this graph are in units of the 
input data set. For this data set, the qualitative review suggests that the data is consistent 
with the theoretical fit, except for the two data points with high cost growth. 
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Figure 37. Q-Q Plot for 44 NASA Completed Missions 


Table 12 provides the recommended NUF based on the historical data. This table 
consists of four types of uncertainty factors: risk not adjusted, conservative, semi- 
aggressive, and aggressive. These factors correspond to three levels of risk: moderate, 
high, and very high. The moderate level of risk is defined as between 10-30% based on 
the data. Additionally, a 30% limit is set because a report to Congress is necessary above 
this level. The high and high plus risk are defined based on the complexity of the 
missions and technology development. They range from 30-70% growth for high and 
higher than 250% for very high. 
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Table 

12. Recommended NASA Uncertainty Factors (NUFs) 

Level of Risk 

NASA Uncertainty Factors 


No- Risk Adjusted 

Conservative 

Semi-Aggressive 

Aggressive 

Moderate (10-30%) 

1 

1.1 

1.2 

1.3 

High (30-75%) 

1 

1.3 

1.5 

1.75 

Very High (>75%) 

1 

1.75 

2.1 

2.5 


The justification for the uncertainty factors for each category is based on actual data 
and from the developed distribution of lognonnal. Most of the data are contained in the 
range from conservative-to-aggressive for the moderate risk level from Table 12. NASA 
missions, such as Lunar Reconnaissance Orbiter (LRO), Wide-Field Infrared Survey 
Explorer (WISE), and Far Ultraviolet Spectroscopic Explorer (FUSE) are at a 10% 
moderate level risk with conservative uncertainty factors. Moreover, Dawn, Orbiting 
Carbon Observatory (OCO) and Wide-field Infrared Explorer (WIRE) are examples of 
the moderate level of risk, but with aggressive uncertainty factors. 

For the high and very high risk levels, there are a wide range of factors (30-70%) 
based on the complexity of the missions. Figure 38 shows NASA missions plotted on the 
lognonnal distribution to provide a clear picture of the data. 
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Figure 38. NASA Missions Data with NASA Developed Lognormal Distribution 

The general guidelines to use the NUFs are: 

- Analyst Judgment 

- Mission Complexity 

- Heritage Level 

- Similarity of Missions 

Finally, the risk factor can be changed as the mission development becomes more 
mature, which allows for the adjusting of the factor to predict the actual cost or cost 
growth. Therefore, a cost risk should not prescribe one method, but rather allow some 
flexibility as long as the analyst gets the information they need to justify their 
methodology. For example, it will be difficult to assess cost risk for a mission at an early 
conceptual stage (which has limited programmatic or technical definition) using the more 
complex methods. For such a case, simple method-using uncertainty factors might be 
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more appropriate and still convey the relative cost risk for the mission. However, a 
mission going through a major milestone should have sufficient detail defined to employ 
a probabilistic method. Given adequate time and trained analysts, it should be feasible to 
use one of the more complex methods. However, if a risk assessment and estimate need 
to be generated very rapidly, then a simpler method must be employed. 

4.2 Validation of NASA Uncertainty Factors 

The first step in the validation process is the selection of the validation data points, 
which were the same as the NASA historical data. Using the developed uncertainty 
factors with NASA data to validate the quality of the factors for several risk levels, a 
small subset of the database was conducted and analyzed, as follows: 

4.2.1 Conservative Factors 

1 . All data with conservative factors with and three risk levels (Figure 39). 

2. Data of conservative factors and missions with moderate risk only (Figure 40). 

3. Data of conservative factors with moderate risk level up to 30% cost growth 
(Figure 41). 

4. Data of conservative factors with high risk level (Figure 42). 
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Figure 39. Applied Conservative NUC to NASA Data 




Figure 40. Applied Conservative NUC for Moderate Risk Level to NASA Data 
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Figure 41. Applied Conservative NUC for Moderate Risk Level for NASA Data up 

to 30% of the Cost Growth 



Figure 42. Applied Conservative NUC for High Risk Level to NASA Data 
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4.2.2 Aggressive Factors 

1. All data on aggressive uncertainty factors with three risk levels and all missions 
(Figure 43). 

2. Data on aggressive uncertainty factors with moderate risk level and all missions 
(Figure 44). 

3. Data on aggressive uncertainty factors with moderate risk level and missions with 
less than 100% cost growth (Figure 45). 
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Figure 43. Applied Aggressive NUC for Three Risk Levels to NASA Data 
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Figure 45. Applied Aggressive NUC for Moderate Risk Level for NASA Missions 

with Less than 100% Cost Growth 
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From the figures above, the aggressive factor for moderate risk level covers 80% of 
the data, thus factors will provide better predictions and estimates for NASA projects. 

Finally, the data analysis should empirically validate previous cost estimates and their 
associated risk analyses. Such a validation would help to improve both data quality and 
risk estimation in NASA missions. It is vital to the credibility of both uncertainty factors 
and its applications in NASA missions. This qualitative method can be valuable for 
providing a better understanding of estimating overall risks for projects. 

4.3 Landsat Data Continuity Mission (LDCM) Estimating Case 

The LDCM is the successor mission to Landsat 7. Landsat satellites have 
continuously acquired multi-spectral images of the global land surface since the launch of 
the Landsat 1 in 1972. The Landsat data archive constitutes the longest record of the land 
surface as viewed from space. The LDCM objective is to extend the ability to detect and 
quantitatively characterize changes on the global land surface at a scale where natural and 
man-made causes of change can be detected and differentiated. It will continue to obtain 
valuable data and imagery to be used in agriculture, education, business, science, and 
government. The LDCM, consistent with U.S. law and government policy, will continue 
the acquisition, archiving, and distribution of moderate-resolution multispectral imagery 
affording global, synoptic, and repetitive coverage of the Earth’s land surface at a scale 
where natural and human-induced changes can be detected, differentiated, characterized, 
and monitored overtime. The science focus area served by LDCM will include: carbon 
cycle, ecosystems, biogeochemistry, and Earth surface and interior. Plan Sub-goal 3A: 
Study Earth from space to advance scientific understanding and meet societal needs. 
NASA and Department of Interior (DOI) U.S. Geological Survey (USGS) were identified 
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as the Landsat Program Management team under the authority of U.S. Code Title 15, 
Chapter 82, “Land Remote Sensing Policy” and Presidential Decision Directive NSTC-3, 
“Land Remote Sensing Strategy.” 

The lead NASA Center for LDCM is the Goddard Space Flight Center (GSFC). The 
lead USGS center for LDCM is the Center for Earth Resources Observation and Science 
(EROS). The LDCM observatory is anticipated to launch aboard an Atlas V Model 401 
launch vehicle from Vandenberg Air Force Base (VAFB) no earlier than December 2012. 
USGS is responsible for the development of the Ground System, excluding procurement 
of the Mission Operations Element (MOE), Flight Operations Team (FOT), and 
establishment of the Mission Operations Center (MOC). The USGS is also responsible 
for LDCM mission operations, after completion of the on-orbit checkout period. NASA 
will serve as the system integrator for LDCM and lead the mission systems engineering 
effort. LDCM is being undertaken by NASA as a stand-alone (‘free-flyer’) mission 
planned for launch as soon as possible to provide continuity of Landsat data. Launch is 
scheduled in December 2012. 

Several independent reviews have been conducted to ensure that the project is in 
compliance with technical, cost and schedule requirements. The LDCM has four main 
elements: Operational Land Imager (OLI), Thennal Infrared Sensor (TIRS), Spacecraft, 
and Ground Operation. Table 13 shows the independent review board qualitative scoring 


of the cost uncertainties for the four main elements. 
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Ta ble 13. Qualitative Cost Uncertainty Rating for the LDCM 


Risk Level 

OL1 

TIRS 

Spacecraft 

Ground Operations 

Moderate 

X 



X 

High 



X 


High + 


X 




This mission completed four independent reviews that assessed the status of the 
mission from the technical and programmatic viewpoint. Table 14 is focused on the 
initial cost for each review. It is evident that the cost growth from System Requirements 
Review/Mission Definition Review (SRR/MDR) to PDR from the project perspective is 
approximately 40%. From the PDR to CDR, the cost growth is 4%. At this point in the 
mission, there is a cost growth on 46% of the mission because of the addition of a 17- 
month schedule. This is a valid increase, as the initial schedule was so unrealistic. 
Additionally, all the TBDs have not yet occurred because the project is still under 
development. 


Table 14. LDCM Life-Cycle Reviews < 

Cost Changes 

LDCM Milestones 

Initial Cost ($M) 

Initial Cost ($M) 
IPAO 

SRR/MDR 

$645 

$888 

PDR 

$904 

$1071 

CDR 

$941 

$1016 

SIR 

TBD 

TBD 
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The technical approach for LDCM refers to any potential technical risk that has a 
known impact associated with cost or schedule and technical uncertainty and growth. 
This includes such things as technology development or inadequate technical margins. If 
the NUFs are used for this project as an aggressive moderate risk with a factor of 1.3, one 
can predict the cost growths will be 30% from the PDR. Thus, total actual cost will be 
$1 175M and will fall in the middle of the lognormal distribution (see Figure 46). 
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Figure 46. LDCM Predicted Cost Growth Four Uncertainty Factors 


4.4 Final Comparison 

The last verification point for the NUF is comparing the four methods of estimating 
the uncertainty factors with the actual NASA data set (Table 15). There are five 
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completed missions that were not used to develop the NUF lognonnal distribution for 
tested and used data sets (see Figure 9). 

Table 15. Comparison of Cost Growth Using Different Uncertainty Factors- Test 


Case 1 

for NUF 




Risk 
Factor as 
Aggressive 
and 

Moderate 
Risk Level 




Mission 

I-Cost 

F-Cost 

NUF(1.3) 

Aerospace(1.816) 

Air 

Force(1.49) 

BAH(1.7) 

TRACE 

$35.6 

$40.3 

$46.28 

$64.65 

$53.04 

$60.52 

SDO 

$520.8 

$633.5 

$ 677.04 

$945.77 

$775.99 

$885.36 

DAWN 

$216.9 

$272.9 

$281.97 

$393.89 

$323.18 

$368.73 

PHOENIX 

$207.0 

$296.5 

$269.10 

$375.91 

$ 308.43 

$351.90 

CLOUDSAT 

$80.2 

$144.0 

$ 104.26 

$145.64 

$119.50 

$136.34 


Figure 47 shows that the NUF is very close to the final cost. Thus, NUF provides a 
better estimate than the others NUFs as displayed in the selected five missions. Note that 
for the Trace, Solar Dynamics Observatory (SDO), and Dawn missions, the NUF are 
within 2% of the final cost growth but, for the Phoenix and CloudSat missions, the final 
cost growth is higher than NUF because it is higher than normal NASA cost growth; 
however, the NUF is closer than other uncertainty factors. These two missions should 
have been classified as a high level of risk and semi-aggressive uncertainty; thus the NUF 
should been 1.5, which yields $310.5 for Phoenix and $120.3 for CloudSat. 
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Figure 47. Estimate Cost Growth with Four Uncertainty Factors Methods 


Finally, using the NUFs for development of initial cost estimating for new P/p, 
which should yield a more realistic estimate and help determine a final actual cost is 
recommended. Furthermore, the gathered data allows one to form an informed a priori 
assessment of future cost growth. Uncertainty in future cost growth is quantified by a 
probability distribution. Arguments based on theory and analyses suggest that the 
lognormal distribution is a reasonable choice. Finally, NUFs provide the parameters for 
the distribution that best fit the NASA cost growth experience. 
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CHAPTER 5 
CONCLUSIONS 

“In today’s environment, hoarding knowledge ultimately erodes your power. If you know 
something every important, the way to get power is by actually sharing it. ” 

Joseph Bardaracco, Jr. 

NASA is a good investment of federal funds and strives to provide the best value to 
the nation. NASA has consistently budgeted to unrealistic cost estimates, whose unreality 
is reflected in the cost growth in many of its programs. NASA has been using available 
uncertainty factors from the Aerospace Corporation, Air Force, and BAH to develop 
projects risk posture. NASA has no insight into the development of these factors and, as 
demonstrated here, this can lead to unrealistic risks in many NASA P/p. This contribution 
of this project is the development of NASA missions’ uncertainty factors from actual 
historical NASA projects in order to estimate cost for independent reviews that provide 
NASA senior management with the information and analysis to determine the appropriate 
decision regarding P/p at KDPs. 

5.1 Summary of Contributions 

This doctoral project has special contributions to cost estimation for NASA P/p and 
specifically for the independent analysis groups that are faced with the challenge of 
providing realistic cost estimates for use in Agency-level decision-making. The 
highlights of contributions are as follows: 

5.1.1 Generated insights into NASA Cost Growth 

This project investigated recent NASA cost growth history for sixty science missions. 
These missions included both Space and Earth Science. These insights are best 
summarized in Chapters 3 and 4. Reasons for cost growth are varied and often poorly 
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understood. At the onset of a Program, technical details are also poorly understood. 
Program Managers provide detailed estimates that lack rigor. It is difficult to push-back 
without some sort of “report card” for the community. This research summarizes the 
reasons for cost growth in many science missions. 

5.1.2 Developed NASA Uncertainty Factors (NUFs) 

This project examined NASA historical cost growth data and developed NUFs to be 
implemented in realistic estimates of probability cost distributions. These NUFs are 
distinctly different from those currently being used in several ways. In Chapter 4, it is 
shown that the NUFs provide some guidelines for cost growth that would be useful in 
many ways. For example, briefing senior management on the magnitudes of cost growth 
typical of NASA projects could be accomplished with Table 12. Furthermore, these 
factors can be used by decision-makers to assess the difference between an independent 
estimate and a Program Manager’s advocacy estimate which is worth reconciling. A 
difference of 10%, for example, might be judged to be insignificant given the amount of 
cost growth experienced in general on all Programs. 

5.1.3 Identified Better-Fitting Cost Distributions for NASA 

The purpose of this research was to develop a risk distribution for NUFs that would 
be applicable in the early stages of the cost estimation of NASA missions. It has been 
found that NASA cost growth fits a lognonnal uncertainty distribution. Coupled with the 
NUFs, the cost risk analysis would produce a more accurate estimate of final costs. This 
is a significant contribution in light of current bias at NASA toward underestimating 
costs. Additionally, the probability distribution of cost growth for sixty NASA missions 
provides evidence of an exponentially-long tail. This is evidence that “Black Swan” 
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programs are not exceptionally rare. It is a challenge to the programmatic and technical 
communities to spot these types of programs and then have to bring that bad news to the 
discussion table. It is recommended that for the programmatic analysts use Figure 34 to 
remind Program Managers that major problems in program execution are not simply a 
rare case of “bad luck.” They happen more often than one would like to admit. 

The cost risk analysis will be better understood because the uncertainty estimating 
will produce a more realistic estimate, in lieu of the signification bias toward 
underestimating that the Agency experiences. As discussed above, this project proved 
that the factors developed are feasible, more relevant to NASA’s missions, and useful for 
estimating the cost risk of future missions. 

5.2 Limitations of NASA Uncertainty Factors 

As with any estimating method, there are limitations to this approach. The NUFs 
implementation has great dependency on the cost analysts for selecting the best range for 
the specific mission. Then, usage of the right factors contains a great deal of uncertainty 
with itself. Additionally, this method is an approximation approach, so some missions 
may fall outside these factors. Finally, expert judgment goes hand in hand with NUF 
usage. One must understand that these factors are based on historical data, which may not 
be relevant in the future due to new manufacturing processes, new technologies, and 
better productivity than the historical data supported. 
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5.3 Future Work 

The following areas can be improved: 

1. The results could be made statistically relevant by simply increasing the number 
of historical missions captured in the database. Any increase in the number of 
missions in the database would result in the increased accuracy of the results. 

2. In addition to increasing the accuracy of the results, studying additional historical 
missions for cost growth and risk data could also be used to provide a good check 
for the methodology that was developed through this project. These results could 
then be checked against actual data for determining where the actual cost of the 
mission is contained within the predicted estimate. 

3. Test the NUF for missions in various life-cycle phases and compare the results to 
this project to detennine if change has any effect on the life-cycles phases on the 
NUF. 

4. Emphasize whether the understanding of the technical and programmatic risk 
during the missions’ reviews will provide more accurate prediction of future cost 
growth for those missions. 

5. The Microsoft® Excel-based tool that was developed for this project is very 
preliminary and basic. A more user-friendly tool should be developed to enable 
the methodology to be used by individuals who are not familiar with the research 
task that developed it. This would greatly enhance the applicability of the factors. 

6. The factors have to be accepted by the cost analysts’ community for 
implementation by the Agency and consider developing an Agency standard to 
request the usage. 
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This research provides an important contribution to the discipline of cost estimating. 
In particular, it has developed a solid database of actual cost growth history and adds 
some statistical rigor to the derivation of cost growth factors based on this data. 
Additionally, it is expected that this work will be referenced for independent cost 
estimates, correction of advocacy-bias in project-generated estimates, and other 
programmatic work. 
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The research project will use mixed-method design, but mostly it is a quantitative 
research approach that seeks prediction for generalization to other NASA projects. The 
qualitative portion will come toward implementation of the developed NUFs. The 
following proposed steps will be conducted to complete this project: 

1 . Literature review of NASA projects cost and schedule growth 

2 . Identification of NASA proj ects 

3. Data selection and analysis 

4. Expert opinion and relevant working testing and evaluation 

5. Develop a method of analysis 

6. Develop NASA uncertainty factors 

7. Test factors 

8. Compare results 

9. Make recommendations to implement the developed process 

10. Publish the work 


1 1 . Complete research project 
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Appendix B 
NASA Cost Risk Policy 

There is no specific cost risk policy that directs the cost estimator on how a cost risk 
assessment should be performed and included in a cost estimate. The only requirement is 
that a cost risk assessment has been conducted, the results incorporated into the estimate 
and the probabilistic cost estimate is presented at the 70% CL. NASA Policy Directives 
(NPDs) are policy statements that describe what NASA must do to achieve its vision, 
mission, and external mandates and who is responsible for carrying out those 
requirements. NASA Procedural Requirements (NPRs) provide Agency-mandatory 
instructions and requirements to implement NASA policy as delineated in an associated 
NPD. The following NPDs and NPRs provide information pertaining to NASA's cost risk 
requirements. These NPRs in conjunction with the Cost Risk volume of the NASA CEH 
provide the guidance and references for the NASA cost estimator to conduct the cost risk 
estimate as appropriate. 

B.l NPR 7120.5 Space Flight Program and Project Management Requirements 

NPR 7120.5, NASA Space Flight Program and Project Management Requirements, 
(http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7120_005D)covers 
requirements by which NASA formulates and implements space flight P/p, consistent 
with the governance model contained in NPD 1000.0, NASA Strategic Management and 
Governance Handbook, (http://nodis.hq.nasa.gov/displayDir.cfm?t=NPD&c=1000&s=0). 

Specific to cost risk, this NPR covers P/p management's cost risk roles and 
responsibilities as well as P/p cost risk requirements by life-cycle phase. This includes: 
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• Risk assessments 

• Risk evaluations 

• Risk mitigations 

• Identification of margin and reserves 

• Associated oversight and approval processes 

A number of cost risk related activities are required early in the project's life-cycle 
(Pre-Phase A through Phase B). Listed below are required activities or products relevant 
to cost risk during a program or project's life-cycle: 

1. A high-level WBS consistent with the NASA standard space flight project WBS, 
schedule, and a rough order of magnitude cost estimate and cost range. 

2. A baseline mission concept document that includes key risk drivers and 
mitigation options. 

3. A preliminary full cost life-cycle cost estimate that includes reserves, along with 
the level of confidence estimate provided by the reserves based on a cost risk 
analysis. 

The instructions and requirements stated in this NPR are associated with the policy 
set forth in NPD 7120.4C, NASA Program/Project Management 
(http://nodis3.gsfc.nasa.gov/ displayDir.cfm?t=NPD&c=7120&s=4C). This document 
describes the management system governing formulation, approval, implementation, and 
evaluation of P/p. 
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B.2 NPR 8000.4 Risk Management Procedural Requirements 

NPR 8000.4, NASA Risk Management Procedural Requirements 
(http://nodis3.gsfc.nasa.gov/ displayDir.cfm?t=NPR&c=8000&s=4) outlines program and 
project requirements and information that pertain to risk management, as required by 
NPR 7120. 5D and NPD 8700.1, NASA Policy for Safety and Mission Success 
(http://nodis.hq.nasa.gov/ displayDir.cfm?t=NPD&c=8700&s=lC). This NPR also 
introduces the continuous risk management (CRM) process and defines risk management 
concepts, risk management requirements, and risk management responsibilities. 

CRM is a 6-step process that is used to manage risk and achieve planned objectives. 
This process involves identifying, analyzing, planning, tracking, controlling, 
documenting, and communicating risks effectively. 

NPR 8000.4 requires P/p to perform risk analyses that consist of estimating the 
likelihood and the consequences of risks and the timeframe in which action must be taken 
on an identified risk to avoid harm. The recommended methods of analyzing risks 
include, but are not limited to, the following: 

• Individual or group expert judgment. 

• Statistical analysis of historical data. 

• Uncertainty analysis of cost, performance, and schedule projections (consists of 
building and running a probabilistic model of the system under investigation, 
including the chance variation inherent in real-life cost, perfonnance, and 


schedule). 
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B.3 Cost Risk Management Requirements in NPR 8000.4 

NPR 8000.4 Chapter 4, "Special Requirements for Programs and Projects," paragraph 
4.2 "Cost Risk Management," requires cost risk management to be part of the CRM 
process and delineates specific cost risk requirements, but does not describe the process 
or how they are to be implemented. This cost estimating handbook contains that 
information. 

B.4 Cost-Risk Management 

While some cost-risk methodologies can be generalized to Space Flight Programs, or 
even non-Space Flight endeavors, the focus and the tools discussed here are applied to 
Categories I & II major Space Flight Projects. The objective of cost risk management is 
to continuously detennine the rolled-up risk impact on the cost of the P/p by organizing, 
obtaining, and using cost-risk infonnation. 

Stakeholder interest in integrated cost-risk was codified in June 2006 with the OMB 
update of Circular A-ll, Part 7 and the Supplement to Part 7 (Capital Programming 
Guide) and in July 2006 with the update of the FAR (FAR Case 2004-019) that 
implements the earned value management system (EVMS) policy in accordance with the 
changes to Circular A-ll, Part 7. These updates require the creation and management of 
risk adjusted budgets. 

This supplemented GAO interest to better NASA cost-risk management as 
documented in the May 2004 GAO report on NASA cost estimating. 

Cost risk management integrates the CRM process, cost estimating, cost-risk 
assessment/analysis (utilizing the identified risks in the project risk list and the cost 
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estimate), and EVM, with procurement, source selection, cost data collection, and cost 
data analysis as supporting disciplines. 

There are three activities that make up integrated cost-risk: Identify and Quantify 
Cost-Risk; Establish Cost-Risk Reporting; and, Manage Cost-Risk Using Reported Data. 
These activities are summarized below: 

Identify and Quantify Cost-Risk 

• Identify and assess risk. 

• Translate risk assessment into cost impact. 

• Perform "S"-curve and CRM scenario-based cost-risk. 

• Incorporate CRM scenario-based and "S"-curve cost-risk in CADRe Part C life- 
cycle cost estimate (LCCE). 

Establish Cost-Risk Reporting 

• Develop RFP CADRe & EVM Data Requirements Description (DRD) and 
equivalent project plan requirements. 

• Evaluate EVM and LCCE DRDs in proposals/project plans. 

• Perfonn Integrated Baseline Review. 

Manage Cost-Risk Using Reported Data 

• Perform EVM performance measurement & CADRe "S"-curve analysis. 

• Compile end-of-contract cost-risk data for database updates, data evaluation, and 
analysis and cost-risk algorithm updates. 


Cost risk management is performed in three overlapping stages during project life- 
cycle phases. Generally speaking, identification, quantification and establishing cost-risk 



106 


reporting occur at the end of each phase, followed by the use of that reporting for cost- 
risk management in the next phase. This cycle repeats as illustrated in Figure 49. 


Program 

Phases 

Formulation 

Implementation 

Flight Projects 
Life Cycle 
Phases 

Pre-Phase A: 
Concept 
Studies 

KDP 

Phase A: 
Concept 
Development 

A KDP 

Phase B: 

Preliminary Design 
B KDP 

Phase C: 
Detailed Design 
C CDR KDP 

Phase D: 
Fab, Assy & Test 

Launch 
D KDP 

Phase E: 
Operations & 
Sustainment 

E KDF 

Phase F: 
Disposal 

F 

Traditional 
Waterfall 
Development 
or Directed 
Missions 


V* 

7 

7 V 

> 

Q* 

An 


AO-Driven 

Projects 

Dot 

Set 

Ste 

5 

5Ct 

pi ^ Sele 

ZAjM 

v 

©3 

V 

— A a 



Legend 

Id & Quantify Cost-Risk 

% 


Establish Cost-Risk Reporting 


> Manage Cost-Risks Using Reported Data 

\7 GPMC Mission Decision Revievv/ICR 
0 CADRe's 


Figure 49. When Integrated Cost-Risk is Required 


Pre-Phase A/Phase A to Phase B 

In pre-Phase A and the early Phase A of formulation, P/p should identify and quantify 
cost-risk to be incorporated in the project's CADRe LCCE that forms the basis for the 
proposed project budget. 

The CADRe has three-parts: Part A - narrative project description; Part B - technical 
characteristics; and, Part C - risk-adjusted LCCE. Part C requires any actual costs-to-date 
plus an estimate -to-complete with cost methodology and cost-risk quantification 
documentation. Near the end of Phase A, an ICE is performed assessing cost and cost risk 
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in preparation for transition to Phase B. CRM risk identification is a key input into cost- 
risk quantification for the project's CADRe LCCE. CRM risk likelihood-based cost 
impacts are compared with the cost estimating cost-risk impacts and reconciled to 
produce the project's CADRe LCCE. Also, in late Phase A, the project develops data 
requirements to establish cost-risk reporting for cost-risk management using the reported 
cost-risk data beginning early and extending throughout Phase B. 

Phase B to Phase C 

In late Phase B, P/p updates their CADRe LCCE including identification and 
quantification of cost-risk and documents reasons for cost growth for the final risk- 
adjusted budget for approval at confirmation. Once approved, the P/p incorporates the 
risk handling budgets for cost-risk in the EVMS's performance measurement baseline 
(PMB) to be tracked and managed in Phase C of implementation. Establishing new cost- 
risk reporting in Phase B is only activated if there are any changes necessary in the 
reporting data used in managing Phase C cost-risk. Projects then incorporate and budget 
risk handling tasks in their EVMS. Projects also flow down the requirements for cost-risk 
in any contractor’s EVMS in all appropriate procurements. 

Phase C to Phases D & E 

The identification, quantification, and updating cost-risk reporting (if necessary) of 
integrated cost-risk is again repeated prior to entry into implementation of Phases D & E 
to manage cost-risk using reported data in those phases. Working synergistically with 
integrated cost risk and the EVM is used to plan and budget for risk handling and 
reporting. P/p offices will also specifically evaluate the EVM cost-risk handling 
performance measurement on a monthly basis. 
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EVMS Control Accounts contain work packages where risk handling activities are 
planned, budgeted, and measured. P/p meeting EVMS requirement thresholds incorporate 
meaningful, measurable, and relevant risk handling activities in the EVMS. Risk handling 
activities are budgeted, scheduled and assessed as part of the project's EVM planning and 
performance assessment process. EVM data is used to track perfonnance measurement 
progress of the risk handling activities, against the project's integrated baseline, that is, 
the PMB integrated with the integrated master schedule (IMS). The rationale for this is 
that all risk handling activities ultimately involve use of project resources (e.g., 
personnel, schedule, and budget). EVM allows the project to plan and assess perfonnance 
based upon an integration of these resources. 

Performance against the plan and EVM reporting can include WBS elements 
identified as risky during integrated cost-risk activities to ensure the Project Manager has 
performance measurement infonnation on those WBS elements most likely to cause cost 
and schedule problems. 

Each month's EAC from the EVMS can include a cost-risk exercise resulting in an 
EAC cost-risk S-curve for the effort. The cost-risk S-curve provides higher quality 
information to the Project Manager about how confident he or she should be about the 
project's EAC versus the contractor's Latest Revised Estimate (LRE) that includes cost 
impacts due to current levels of risk. Using EVM metrics (e.g., Cost Perfonnance Index 
(CPI); Schedule Perfonnance Index (SPI); Schedule/Cost Index (SCI); etc.) in 
combination with Microsoft® Excel and Monte Carlo simulation software, Control 
Account and Work Package activity cost-risks can be modeled and statistically 


summarized for the S-curve evaluation. 
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EVM cost-risk reporting requirements should be described in the solicitation's data 
requirements section such that contractors understand that risks identified in the cost 
estimate, by the source evaluation boards and independent risk identification teams are to 
be reported in the EVM contract perfonnance reports (CPR). Such CPR data requirement 
language should read like the following as developed by the EVM Working Group and 
posted on the Cost Analysis Division website. 

Contents 

The CPR shall include data pertaining to all authorized contract work, including both 
priced and unpriced efforts that have been authorized at a not-to-exceed amount in 
accordance with the Contracting Officer's direction. The CPR shall separate direct and 
indirect costs and identify elements of cost for all direct reporting. The CPR shall include 
Formats 1 through 5, down to a WBS Level -4. A lower level of reporting may be 
required for elements that are classified as "special interest" technical, schedule, or cost 
risk areas. 

Earned value performance measurement data for government and/or contractor- 
identified medium- and high-risk WBS items shall be reported on Fonnat 1 of the 
monthly CPR until such time as both government project management and the contractor 
agree that they no longer represent high risks. This reporting shall be at a level where the 
risk resides in the WBS. For medium- and high-risk elements lower than Level 4, specific 
narrative variance analyses are not required unless classified as "special interest". 

To ensure an integrated approach to risk management, the data provided by this CPR 
DID shall be in consonance with the WBS, Integrated Master Schedule (IMS), Risk 
Management Processes, Plans and Reports (where required), Probabilistic Risk 
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Assessment Processes and Reports (where required), the CADRe and the 
Monthly/Quarterly Contractor Financial Management Reports (533/Q). The Financial 
Management Reports shall include reconciliation between the 533Q and the CPR. This 
reconciliation may be included within the required CPR formats. 

Format 

CPR formats shall be completed according to the instructions outlined in DI-MGMT- 
81466A and the following forms: Format 1 (DD Form 2734/1); Format 2 (DD Form 
2734/2); Format 3 (DD Form 2734/3); Format 4 (DD Form 2734/4); and Format 5 (DD 
Form 2734/5). Samples of the forms are located at 
http://www.dtic.mil/whs/directives/infomgt/forms/ddforms2500-2999.htm. Variance 
analysis thresholds which, if exceeded, require problem analysis, narrative explanations 
and corrective action plan descriptions for all level three and other special interest WBS 
elements. Variance analysis thresholds will initially be +/- 10% of both current and 
cumulative cost and schedule variance to date. The variance analysis thresholds may 
change once the personnel evaluate the contractor's schedule and cost performance and 
risk. 

Special emphasis should be placed in the variance analysis on cost and schedule 
growth linked to technical risks (e.g., technology development efforts, design 
engineering, integration, complexity, project management, systems engineering, duration 
constraints, etc.) identified by both the government and contractor. 

Contractor format may be substituted for CPR formats whenever they contain all the 
required data elements at the specified reporting levels in a form suitable for NASA 
management use. The CPR shall be submitted electronically and followed up with a 
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signed paper copy. The American National Standards Institute (ANSI) X12/XML 
standards (transaction sets 839 for cost and 806 for schedule), or the United National 
Electronic Data Interchange for Administration, Commerce and Transport (EDIFACT), 
http://www.unece.org/trade/untdid/welcome.htm equivalent, or any other electronic 
delivery method deemed acceptable by the Project Office shall be used for Electronic 
Data Interchange. 

Refer to the EVM website, http://evm.nasa.gov, for additional information regarding 
EVM. Refer to NPR 7120.5 for EVM applicability and NASA requirements. 
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Appendix C 

NASA Cost Risk as Part of the Cost Estimating Process 

Cost risks are those risks due to economic factors such as rate uncertainties, cost 
estimating errors, and statistical uncertainty inherent in the estimate. Cost risk is 
dependent upon other fundamental risk dimensions (technical, schedule, and 
programmatic risks) so these must all be assessed to arrive at a true picture of project 
risk. 

Cost-risk assessment takes into account cost, schedule, and technical risks that are 
then factored back into the cost estimate. To quantify the cost impacts due to risk, sources 
of risk need to be identified. NASA cost analysts should be concerned with three sources 
of risk and ensure that the model calculating the cost accounts for: 

• Risk inherent in the cost estimating methodology. For example, if a regression- 
based cost estimating relationship (CER) is used, it has an associated standard 
error of the estimate (SEE), confidence intervals, and prediction intervals, any of 
which can be used to include cost estimating methodology risk in the estimate. 

• Risk inherent in the technical aspects of the systems being developed. Into this 
category of risk fall risk sources such as the technology's state-of-the-art 
design/engineering (Technology Readiness Levels (TRLs)) are good indicators of 
this risk source), integration, manufacturing, schedule, complexity, etc. 
Quantifying the cost impacts due to these kinds of risk is not as statistically 
derivative as is CER risk. Figure 50 graphically displays the effects of cost 
estimating methodology risk and technical input risk. 
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• Risk inherent in the correlation between WBS elements. Correlation 
assessment determines to what degree one WBS element's change in cost is 
related to another’s and in which direction. For example, if the cost of the 
satellite's payload goes up and the cost of the propulsion system goes up then 
there is a positive correlation between both subsystems' costs. Many WBS 
elements within space systems have positive correlations with each other and the 
cumulative effect of this positive correlation tends to increase the range of the 
possible costs. 

Even as early as Pre-Phase A, it is important to capture risk in cost estimates, 
especially technical, schedule, programmatic and cost data. Even at this early stage, there 
are many risks that can and should be identified and addressed in a cost risk assessment. 
Cost estimating uncertainty, technical input variable uncertainty, and correlation risks all 
need to be considered. Schedule risk can be handled outside these three types of risk by 
applying probabilistic activity duration risk to the critical path analysis (CPA). 
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Figure 50. Cost Modeling and Technical Input Risk 


Working with project office staff, the cost estimator should identify cost-risk drivers 
and vary the operating scenarios and input parameters through the conduct of 
comprehensive probabilistic and deterministic cost-risk and sensitivity analyses. It is the 
job of the cost estimator to estimate the effects of identifying, assessing, and analyzing 
cost-risk drivers (e.g., probabilistic cost-risk analysis) and varying cost drivers (e.g., 
deterministic cost-risk) and to revise the LCC estimates reflecting the selected variations, 
pointing out the relationship between the LCC and the key technical and/or operational 
parameter risks. Discrete technical cost-risk assessments involve identifying and cost 
estimating specific cost-driving technical risks. 

For example, a notional new electronic component for a spacecraft might have risk in 
key engineering performance parameters (KEPPs) such as dynamic load resistance, 
operating voltage, power regulation, radiation resistance, emissivity, component mass, 
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operating temperature range and operating efficiency. Technical staff can identify these 
KEPP risks during cost-risk assessment. Instead of probabilistic distributions and Monte 
Carlo simulations, however, mitigation costs for these risks are estimated based on their 
probabilities of manifesting discrete changes in the technical parameters (e.g., increased 
component mass or power regulation). Justifying the amount of cost risk dollars is a 
function of the detail specification of cost estimating, technical, and correlation risks that 
drive the cost risk range. Cost risk dollars that add, for example, 30% additional costs to 
the point estimate, have to be defensible with a cost-risk methodology that justifies the 
endpoints of individual WBS element cost-risk distributions, SEE regression line, and 
solid correlation coefficients. 

As a project moves through the conceptual design phase, the range of feasible 
alternatives decrease and the definition of those alternatives increase. At this stage, there 
is a crucial need to identify pertinent cost issues and to correct them before corrective 
costs become prohibitive. Issues and cost drivers must be identified to build successful 
options. By accomplishing a cost estimate on proposed project alternatives, a Project 
Office can detennine the cost impact of the alternatives. These cost drivers feed an 
increasingly detailed cost-risk assessment that takes into account cost, technical, and 
schedule risks for the estimate. The point estimate and the risk assessment work together 
to create the total LCC estimate. 

As a project moves through the preliminary design phase and the project definition 
increases, cost estimators should keep the estimate up-to-date with definition changes and 
have a full cost risk assessment to defend the estimate, reduce updated estimate turn- 
around time, and give the decision-maker a clearer picture for "what if' drills or major 
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decisions. The role of the cost estimator during this phase is critical. It is important to 
understand the basis of the estimate, from the technical baseline to the cost risk 
assessment and to be able to document and present the results of these efforts to the 
decision-makers. It is the cost estimator's responsibility to ensure the best possible LCCE 
with recommended levels of unallocated future expense (UFE) based on updated cost risk 
assessments in Phase B. These estimates will support budget fonnulation and source 
selection support in the transition from Phase B to Phases C/D. 

When conducting Phase C/D estimates, new information collected from contractor 
sources and from testing must be fed back into the point estimate and the risk assessment, 
creating a more detailed project estimate. During this phase, the cost-risk assessment 
should be very detailed, not only including any changes in requirements or project 
design, but other details provided by project technical experts such as testing and 
schedule impacts. While the product is being designed, developed, and tested, there are 
changes which can impact the estimate and the risk assessment. It is critical to capture 
these changes to maintain a realistic program estimate now and in the future. During this 
phase, programmatic data may have just as much of an impact on the estimate and risk 


assessment as technical data. 
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The Twelve Tenets of NASA Cost-Risk 
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Tenet 1: NASA cost- risk assessment, a subset of cost estimating, supports cost 
management for optimum project management. 

Tenet 2: NASA cost-risk assessment is based on a common set of risk and uncertainty 
definitions. 

Tenet 3: NASA cost- risk assessment is a joint activity between subject matter experts 
and cost analysis. 

Tenet 4: NASA cost-risk is composed of CERs and technical risk assessment plus cost 
element correlation assessment influenced by other programmatic risk factors. 

Tenet 5: NASA technical cost-risk assessment combines both probabilistic and discrete 
technical risk assessments. 

Tenet 6: NASA cost-risk probability distribution is justifiable and correlation levels are 
based on actual cost history to the maximum extent possible. 

Tenet 7: NASA cost-risk assessment ensures cost estimates are likely-to-be-vice as 
specified for optimum. 

Tenet 8: NASA cost-risk assessments account for all known variance sources and include 
provisions for uncertainty. 

Tenet 9: NASA cost-risk can be an input to every cost estimate’s cost readiness level 
(CRL). 

Tenet 10: NASA cost-risk integrates the quantification of cost-risk and schedule risk by 
enlisting the support of NASA schedule and EVM analysts. 

Tenet 11: NASA decision-makers need to know how much money is in the estimate to 
cover risk events, which WBS elements are allocated, and the CL of the estimate. 
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Tenet 12: NASA project cost-risk data, collected as a function of government and 
contractor project estimates and actual, contract negotiation and contract DRDs, is 
compiled into the OCE database. 
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Appendix E 

Relationship of the Research Project and Published Literature 

This section identifies key publications related to cost and schedule growth and offers 
an assessment of this literature in relation to the research project. 

Research Problem Statement: 

The research proposes to develop uncertainty factors from actual NASA historical 
project data to be used to classify risk for future cost estimation and support the 
independent reviews which inform NASA senior management and enable them to make 
the right decision regarding the project progress. 

Problem Area 1 

Determine NASA projects from which to gather data from as it relates to cost and schedule 
growth for science missions. 

Problem Area 2 

Develop a method to evaluate NASA project cost and schedule data by evaluating causes for 
growth and create measurement formalisms that account for multiple sources of growths. 

Problem Area 3 

Develop NASA Uncertainty Factors by capturing the trend of growth data from the selected 
science missions and compare these factors with other uncertainty factors. 

Problem Area 4 

Bring together research and uncertainty factors developed in problem area one through three into 
a coherent tool to be use in quantification risk for NASA future projects. 

Table 16 presents an assessment of the literature with respect to these 4 problem 

areas. A color-coding scheme was defined and presented below. The color code indicates 

the degree to which the problem area is addressed in the referenced work. 


COLOR CODING SCHEME 

Red: Problem area not addressed in the referenced article or work. 

Yellow: Problem area addressed to some extent in the referenced article or work; but, insufficient 
to meet this dissertation’s research objectives. 

Green: Problem area addressed in the referenced article or work. 
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Table 16. Literature Assessment Using the Four Problem Areas 


Literature Assessment 

Problem 
Area 1 

Problem 
Area 2 

Problem 
Area 3 

Problem 
Area 4 

Arena, V. A, and et al. (2006). “Impossib 
Published 2006. by the RAND Corporati 

e certainty: cost risk analysis for Air Force systems.” 
on. 



Bitten, R.E., Bearden, D. A., Lao, L. Y., Park, T.H. (2003). “The Effect of Schedule constraints 
on the Success of Planetary Missions.” 2003 Elservier Ltd. 



Bitten, R.E., Emmons, D.L, and Freaner, C.W. (2008). “In Search of the Optimal Funding 
Profile: the Effect of Funding Profiles on Cost and Schedule Growth.” ISPA/SCEA 2008 Joint 
International Conference. The Netherlands May2008. 



Bitten, R.E., Emmons, D.L., and Freaner, C.W. (2005). “Using Flistorical NASA Cost and 
Schedule Growth to set Future Program and Project Reserve Guidelines.” 



Connley, Warren. (2004). “Integrated Risk Management within NASA Programs/Projects, ” 
GSFC. 



Coonce, T. (2008). “NASA Cost and Risk Workshop, ” September 2008. 



Cooper, L. (2003). “Assessing Risk from a Stakeholder Perspective, ” IEEE, Paper# 1078. 



Datta, S., and Mukherjee, S. (2001). “Developing Risk Management Matrix for Effective Project 
Planning -An Empirical Study, ” by the Project Management Institution, Vol. 32, No. 2, pp 45-57. 



Dillon, Robin. (2003). “ Programmatic Risk Analysis for Critical Engineering Systems under 
Tight Resource Constraints ,” Operations Research, INFORMS, vol. 51, No. 3, May-June 2003, 
pp. 354 - 370. 



Emmons, D.L., Bitten, R.E., and Freaner, C.W. (2006). “Using Flistorical NASA Cost and 
Schedule Growth to Set Future Program and Project Reserve Guidelines.” IEEE Paper #1545. 
December. 
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Literature Assessment 

Problem 

Problem 

Problem 

Problem 


Area 1 

Area 2 

Area 3 

Area 4 


GAO: Government Accountability Office, (2001). “Major Management Challenges and Program 
Risks for NASA, ” report issued January 2001 . 



Grey, S. (1995). Practical Risk Assessment for Project Management. By John Wiley & Sons. 
Hulett, D. (2007). “Integrated Cost/Schedule Risk Analysis, ” Program Management Challenge, 
2007. 



Hulett, D. (2007). “Integrated Cost/Schedule Risk Analysis, ” Program Management Challenge, 
2007. 



Jiang, P., Haimes, Y. Y., (2004). “Risk Management for Leontief-Based Interdependent 
Systems,” Risk Analysis, Vol. 24, No. 5. 



Kaplan, S., 1997. “The Words of Risk Analysis,” Risk Analysis, Vol. 4, No. 17. 



Kellogg, R., Phan, S. (2002). “ An Analogy-Based Method for Estimating the Costs of Space- 
Based Instruments,” IEEEAC Paper #1160. 



Mlynczak, B., Perry, B., Science Support Office, NASA. “SMD Earth and Space Mission Cost 
Driver Comparison Study.” March 2009. 


Nadler, David. (2004). “ Building Better Boards, ” Harvard Business Review. 


NASA Independent Program Assessment Office-Standard Operating Procedure, 2008. 


NASA- NPR 7120.5 Space Flight Program and Project Management Requirement, March 6, 
2007. 



NASA-NPR 8000.4, issued on April 2002. 



National Research Council. 2010. Controlling Cost Growth of NASA Earth and Space Science 
Missions. National Academy Press, Washington, D.C. 








122 


Literature Assessment 

Problem 

Problem 

Problem 

Problem 


Area 1 

Area 2 

Area 3 

Area 4 


NASA 2008. “SMD Cost/Schedule Performance Study- Summary Overview.” Presentation by B. 
Perry and C. Bruno, NASA Science Support Office; M. Jacobs, M. Doyle, S. Hayes, M. Stancati, 
W. Richie, and J. Rogers. Science Applications International Corporation. January 2008. 



Parsons, V.S. (2007). Searching for “Unknown Unknowns. ” Engineering Management Journal, 
Vol. 19, No.l. 



Pinto, C. A., Arora, A., Hall, D., Ramsey, D., Telang, R., 2004. “Measuring the Risk-Based 
Value of IT Security Solutions”, IEEE IT Professional, v.6 no. 6, pp. 35-42. 



Pinto, C. A., Arora, A., Hall. D., Schmitz, E., 2006. “Challenges to Sustainable Risk 
Management: Case Example in Information Network Security,” Engineering Management. 



Ruckelshaus, William D., “Risk, Science, and Democracy,” Issues in Science 
and Technology, Vol. 1, No. 3, 1985, pp. 19-38. 



Schuyler, J. (2001). Risk and Decision Analysis in Projects, Library of Congress Cataloging in 
publication data. 



Smart, C. (2007). “Cost and Schedule Interrelationships ,” NASA Cost Analysis Symposium, 
Vol. 14, No. 5. 




123 


Appendix F 

Non-NASA Uncertainty Factors 

There are three different sets of uncertainty factors, below: 

1. Uncertainty factors that were developed by Booz- Allen and Hampton 
from NRO missions. 


Table 17. NRO Missions Uncertainty Factors 


Level 

Aggressive 

Most Likely 

Conservative 

Very Low 

N (1.2, 0.05) 

N (1.00, 0.05) 

N (0.8, 0.05) 

Low 

N (1.2, 0.15) 

N (1.00, 0.15) 

N (0.8,0.15) 

Moderate 

N (1.2, 0.25) 

N (1.00, 0.25) 

N (0.8,0.25) 

High 

N (1.2, 0.35) 

N (1.00, 0.35) 

N (0.8,0.35) 

Very High 

N (1.2, 0.45) 

N (1,00, 0.45) 

N (0.8,0.45) 


2. Uncertainty factors that were developed from Air Force missions. 


Table 18. Air Force Missions Uncertainty Factors 


Level 

Optimistic 

Most Likely 

Pessimistic 

Low 

0.95 

1 

1.1 

Low Plus 

0.96 

1 

1.23 

Moderate 

0.97 

1 

1.36 

Moderate Plus 

0.98 

1 

1.49 

High 

0.98 

1 

1.61 

High Plus 

0.99 

1 

1.74 

Very High 

1 

1 

1.87 

Very High Plus 

1 

1 

2 


3. 


Uncertainty factors that were developed by Aerospace Corporation. 



Table 19. The Aerospace Corporation Uncertainty Factors 


Level 

Low 10% 

Mid 50% 

High 90% 

SEE 

Low 

0.977 

1 

1.117 

0.05 

Medium 

Low 

0.932 

1 

1.35 

0.15 

Medium 

0.887 

1 

1.583 

0.25 

Medium 

High 

0.842 

1 

1.816 

0.35 

High 

0.797 

1 

2.05 

0.45 

High+ 

0.752 

1 

2.5 


Very High 

0.707 

1 

3 


Very High 
+ 

0.662 

1 

4.5 


Extra 

High 

0.617 

1 

6 
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Appendix G 

Goodness of Fit Summary for All Tested Cases 



Lognarnr«l 

Bur 

— Lo^Aogafte 

— Bm 

— Gan &*«rr» vat* 

— Hcrrr+l 

P«0r>cr © 







# 

Distribution 

Kolmogorov 

Anderson 

Smirnov 

Darling 

Statistic 

Rank 

Statistic 

Rank 

l 

Beta 

0.171151 

8 

4.05819 

7 

2 

Burr 

0.060433 

1 

2.09441 

5 

3 

Gen. Extreme Value 

0.116518 

5 

0.414936 

1 

4 

log-Loeistic 

0.168776 

6 

4.93337 

8 

5 

Lognormal 

0.0952 

2 

3.37255 

4 

6 

Normal 

0.169327 

7 

1.26063 

2 

1 

Pearson 6 

0.060625 

4 

2.10491 

2 

8 

Weibull 

0.065478 

3 

2.03035 

3 


Figure 51. PDF and the Goodness of Fit of 39 Missions 
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Figure 52. PDF and the Goodness of Fit of 44 Missions 
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Figure 53. PDF and the Goodness of Fit of 50 Missions 
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Figure 54. PDF and the Goodness of Fit of 52 Missions 
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Figure 55. PDF and the Goodness of Fit of 54 Missions 
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Appendix H 

NASA Lognormal Distribution Model 


NASA UF For Log-normal Distribution for Amer*s Project 
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Figure 56. NASA Lognormal Distribution Model 
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NASA Uncertainty Factors 
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The NUFs listed below were developed as part of this research paper. This table 
should be used with the NASA lognormal model from Appendix H. 


Tab! 

le 20. NUFs Developed from this Research Project 

Level of Risk 

NASA Uncertainty Factors 


None - Risk 
Adjusted 

Conservative 

Semi-Aggressive 

Aggressive 

Moderate (10-30%) 

1 


1.2 

1.3 

High (30-75%) 

1 

mm 

1.5 

1.75 

Very High (>75%) 

1 

1.75 

2.1 

2.5 






Appendix J 

NASA Joint Confidence Level Paradox - A History of Denial 
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Historical Cost and Schedule Growth Data Set 

Compiled Cost & Schedule Growth Data Set 

The following eosl and schedule growth data is a combined list of the earliest available and latest 
available data for 188 projects. Some of the names are the same, but supplementary data led us to 
believe they were separate projects. In fact they may not be. renaming, rebaselineing. and whitewashing 
make this type of data mining and analysis very difficult. All data comes from reputable sources, 
however errors probably exist, and some projects are still in development so values may continue to 
evolve. 


Page 105 4/17/2009 
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NASA V Joint Confidence Level Paradox - A History of Denial 


2009 Cost Estimating Symposium 


It shows an average cost growth of 98.2%, a median cost growth of 53.3%, and average schedule growth 
of 56.8%, and a median schedule growth of 34.9%. This is abysmal to say the least and exceeds many of 
the recently published papers values. I'hLs data was not obtained until alter paper w as completed, so 
none was used in our analy sis, but Is included in an attempt to aide future researchers. 


Them© 

Name 

Initial 

Latest 

Available 

Percent 

change 

Initial 

Schedule 

Final 

Schedule 

Percent 

Change 

Heliophysics 

ACE 

$141.10 

$ 108.50 

-23.1% 

57 

62 

8.2% 

Earth Sci 

ACRIMSAT 




43 

45 

4.7% 


ACTS 

$ 354.00 

S 656.00 

853% 

48 

98 

104.2% 


AFE 

$ 159.00 

$ 387.00 

143.4% 

Canceled 




AHMS 

$ 55.00 

$ 55.00 

0.0% 




Heliophysics 

AIM 

$ 61.10 

$ 81.20 

32.9% 

40 

50 

24.7% 


Apollo 

$7,000.00 

$ 25,400.00 

262.9% 




Earth Sci 

Aqua 

$ 762.50 

$ 1,006.00 

31.9% 

89 

107 

19.4% 

Earth Sci 

Aquarius 




50 

69 

38.0% 

Manned 

ASRM 

$1,506.70 

$ 3,251.80 

115.8% 

Canceled 



Manned 

ATP 

$ 372.00 

$ 1,053.00 

183.1% 





AURA 

Aura (Chem-1) or 

$ 524.00 

$ 763.00 

45.6% 

114 

133 

17.4% 

Earth Sci 

Chemistry 

AXAF 

$1,410.00 

$ 6,022.00 

327.1% 

41 

60 

46.3% 

Heliophysics 

BARREL 
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Earth Sci 

CALIPSO 

$ 68.20 

$ 170.3 

149.7% 

38 

89 

135.4% 

Planetary 

Cassini 

$1,436.40 

$ 1,375.90 

-4.2% 

92 

110 

19.6% 

Manned 

CAU 

$ 442.00 

$ 454.00 

2.7% 




ASO 

Chandra 




69 

79 

14.5% 

ASO 

CHIPSAT 




30 

40 

33.3% 

Heliophysics 

CINDI 




41 

95 

131.7% 

Manned 

CLCS 

Clementine 

$ 175.00 

$ 399.00 

128.0% 

Canceled 

19 


Earth Sci 

Cloud Sat 

$ 80.20 

$ 144.00 

79.6% 

36 

85 

136.6% 


Cluster 

Cluster-2 (Rumba & 




75 

81 

8.0% 

Heliophysics 

Tengo) 

Cluster-2 (Salsa & 
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Heliophysics 

Samba) 




75 

130 

73.3% 

ASO 

COBE 




68 

88 

29.4% 

Planetary 

CONTOUR 

$ 69.10 

$ 96.80 

40.1% 

38 

40 

4.2% 


Earth Sci 
Planetary 
Planetary 
Earth Sci 


Cosmic Background 

Explorer 

COSTR 

CRAF 

CRRES 

DART 

DAWN 

Deep Impact 

Deep Space 1 

DSMS 

EO-1 

ERAST 

ESSP 


$97.50 $ 159.70 63.8% 

$221.00 $673.00 204.5% 

$3,593.00 $3,351.00 -6.7% 


202.8 287.1 41.6% 

$194.10 $252.00 29.8% 

$ 73.30 $ 99.30 35.5% 


$ 72.00 
$ 181.30 
$145.10 


$ 158.1 
$ 173.00 
$ 171.80 


Canceled/ Development 


38 

86 

126.3% 

31 

43 

38.7% 

37 

53 

43.2% 

44 

63 

43.9% 

33 

43 

28.5% 


36 


33 

58 

74.2% 
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119.6% 

-4.6% 

18.4% 


4/17/2009 
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NASA V Joint Confidence Level Paradox - A History of Denial 


2009 Cost Estimating Symposium 



ET 

$ 349.60 

$961.70 

175.1% 




ASO 

EUVE 

$ 107.40 

$ 322.00 

199.8% 

48 

58 

208% 

Heliophysics 

FAST 

$ 32.50 

$ 42.90 

32.0% 

44 

89 

101.1% 


FCF 

$118.90 

$114.10 

-4.0% 





FTS 

$ 317.00 

$ 485.00 

530% 

Canceled 




FTS 

$ 317.00 

S 453.20 

43.0% 




ASO 

FUSE 

S 85 90 

$143.7 

67.3% 

44 

107 

143 2% 

ASO 

GALEX 

$41.10 

$87.10 

111.9% 

37 

60 

61.4% 

Planetary 

Galileo Orbiter 




43 

136 

216 3% 

Planetary 

Galileo Probe 




43 

136 

216.3% 

Planetary 

Galilleo 

$ 276.20 

$ 1,639.00 

493.4% 




Earth Sci 

Genesis 

$126.10 

$ 162.90 

29.2% 

34 

46 

34.9% 

Heliophysics 

Geospace RBSP 





44 


Heliophysics 

Geotail 




58 

58 

0.0% 


GGS 

$ 334.00 

$ 649.00 

94.3% 




ASO 

GLAST 




66 

93 

40.9% 

Earth Sci 

Glory 




88 

106 

20.5% 


GOES 

$ 554.60 

$ 1,241.00 

123.8% 





GOES 

$691.00 

$ 1,787.00 

158.6% 




Earth Sci 

GOES 1 




45 

102 

126.7% 

Earth Sci 

GOES J 




53 

115 

117.0% 

Earth Sci 

GOESK 




57 

138 

142.1% 

Earth Sci 

GOES L 




65 

175 

169.2% 

Earth Sci 

GOES M 




74 

189 

155.4% 

Earth Sci 

GOES N 




41 

96 

134.1% 

Earth Sci 

GOESO 




47 

126 

168.1% 

Earth Sci 

GOES P 




83 

137 

65.1% 

ASO 

GP-B 

$ 351.00 

$ 709.30 

102.1% 

74 

128 

72.7% 

Earth Sci 

GPM 




113 

149 

31.9% 

Earth Sci 

GRACE 

$ 79.30 

$ 88.40 

11.5% 

42 

60 

43.8% 

Planetary 

GRAIL 




8 

44 

450.0% 


GRO 

$ 183.80 

$ 677.00 

268.3% 





GRO - (Compton 







ASO 

Gamma Ray Ob) 




62 

127 

104.8% 


HESSI 

$ 32.00 

$ 63.50 

98.4% 

31 

51 

61.9% 

ASO 

HETE 

$8.40 

$ 23.50 

179.8% 

37 

49 

34.2% 


HST 

$ 435.00 

$ 1,682.00 

286.7% 




ASO 

HSTSI 




64 

140 

118.8% 

ASO 

HST SSM 




70 

146 

108.6% 

ASO 

HST-OTA 

$ 115.80 

$561.70 

385.1% 

70 

146 

108.6% 

Heliophysics 

IBEX 




33 

36 

9.1% 

Earth Sci 

ICESAT 

$ 121.30 

$ 177.00 

45.9% 

43 

73 

70.7% 

Heliophysics 

IMAGE 

$ 83.60 

$ 89.20 

6.7% 

42 

49 

15.5% 

ASO 

INTEGRAL 

$ 8.20 

$ 11.90 

45.1% 




Earth Sci 

JASON 

$ 77.50 

$ 87.80 

13.3% 

70 

94 

34.3% 

Planetary 

JUNO 




51 

51 

0.0% 

ASO 

JWST 

$ 900.00 

$ 4,900.00 

444.4% 

92 

114 

23.9% 

ASO 

Kepler 




48 

64 

33.3% 


LADEE 





38 


Earth Sci 

Landsat 7 

$387.10 

$ 508.80 

31.4% 

55 

80 

44.7% 


LANDSAT-D 

$260.10 

$ 538.00 

1 06.8% 
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LCROSS 




27 

28 

3.7% 

Earth Sci 

LDCM 




38 

38 

0.0% 

ASO 

USA 




55 

120 

118.2% 

SSE 

LRO 




38 

39 

2 6% 

Manned 

LTMCC 

S 87.10 

$ 76.90 

-11.7% 




Planetary 

Lunar 

$56.20 

$ 56.60 

0.7% 

30 

34 

10.7% 

Planetary 

MAGELLAN 

$ 322 80 

S 856.00 

165.2% 

61 

73 

19 7% 


MAP 

$ 88.30 

$ 94.20 

6.7% 

56 

64 

14.4% 

ASO 

MAPorWMAP 




54 

61 

13.0% 

Planetary 

MPL 




45 

46 

2.2% 

Planetary 

MCO 

$ 183.60 

$ 189.70 

3.3% 

39 

45 

16.6% 


MEDS 

$ 201.70 

$210.10 

4.2% 




Planetary 

MER 

$ 499.40 

$ 767.00 

53.6% 

34 

34 

1.1% 


MER-A or MER03 - 







Planetary 

SPIRIT) 




35 

35 

0.0% 

Planetary 

MER-B (Opportunity) 




35 

36 

2.9% 

Manned 

MERCURY 

$ 196.92 

$ 384.00 

95.0% 




Planetary 

Messenger 

$ 191.50 

$ 288.70 

50.8% 

46 

57 

23.7% 

Planetary 

MGS 

$ 140.20 

$ 130.70 

-6.8% 

31 

34 

8.7% 

Planetary 

MMM 




31 

35 

12.9% 

Heliophysics 

MMS 




85 

85 

0.0% 

Planetary 

MRO 

$ 334.40 

$ 450.0 

34.6% 

43 

47 

9.4% 

Planetary 

MSL 

$ 650.00 

$2,300.00 

253.8% 

45 



Planetary 

NEAR 

$ 150.00 

$ 124.90 

-16.7% 

29 

29 

0.0% 

Planetary 

New Horizons 




44 

44 

0.0% 


NMP 

$ 111.70 

$ 176.40 

57.9% 




Earth Sci 

NPP 




79 

123 

55.7% 


NSCAT 

$ 100.40 

$ 255.00 

154.0% 




ASO 

NuStar 





37 


Planetary 

Mars Observer 

$ 306.00 

$ 994.00 

224.8% 

100 

126 

26.0% 

Earth Sci 

OCO 




52 

67 

28.8% 

Planetary 

ODYSSEY 

$ 267.20 

$366.10 

37.0% 

33 

34 

3.0% 

Manned 

OMV 

$ 236.00 

$814.00 

244.9% 

Canceled 



Earth Sci 

OSTM 




72 

74 

2.8% 

Planetary 

PATHFINDER 

$ 150.00 

$ 174.20 

16.1 % 

38 

49 

30.6% 

Planetary 

Phoenix/Scout 7 




41 

41 

0.0% 

Earth Sci 

QUICKSCAT 




14 

21 

50.0% 

Heliophysics 

RHESSIwasHESSI 




31 

53 

71.0% 


Rosetta 

$ 28.40 

$ 40.10 

41.2% 




Heliophysics 

Sampex 




37 

38 

2.7% 

Heliophysics 

SDO 




52 

68 

30.8% 


Sea Winds 

$ 130.20 

$ 148.80 

14.3% 




Heliophysics 

SET-1 




39 

89 

128.2% 

ASO 

SIM 




73 

191 

161.6% 

Manned 

SLWT 

$ 172.50 

$ 129.00 

-25.2% 




Heliophysics 

SNOE 




23 

35 

52.2% 

ASO 

SOFIA 

$ 234.80 

$ 840.00 

257.8% 

48 

207 

331 .3% 

Heliophysics 

SoHO 




63 

72 

14.3% 


Solar Orbiter 





84 


Heliophysics 

Solar B or HINODE 

$ 99.30 

$ 80.40 

-19.0% 

58 

83 

43.1% 

Earth Sci 

SORCE 

$ 68.00 

$ 74.50 

9.6% 

32 

55 

71.6% 
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Manned 

Shuttle 

$5,800.00 

$17,789.00 

206.7% 




Manned 

Shuttle - With Reserves 

$6,960.00 

$17,789.00 

155.6% 




Manned 

Shuttle - Endeavor 

$2,100.00 

$1,800.00 

-14.3% 




Manned 

SSME 

$1,267.10 

$3,051.50 

140.8% 




Manned 

Space Station 

$9,446.24 

$ 45,000.00 

3764% 




ASO 

SIRTF or Spitzer 

$ 472.00 

$712.00 

50.8% 

55 

88 

60.1% 

Manned 

SRM 

$ 338.60 

$ 706.70 

1087% 




Manned 

SS 

$25.12 

$ 28 94 

15.2% 




Heliophysics 

ST-5 

$26 30 

48.7 

852% 

48 

78 

62.5% 

Heliophysics 

ST-6 





56 


ASO 

ST-7 




47 

69 

89.4% 

Heliophysics 

ST-8 




37 

32 

-13.5% 

CT 

ST-9 




42 

42 

0.0% 

Planetary 

STARDUST 

$ 117.80 

$ 126.4 

7.3% 

35 

39 

10.9% 


STDRS 

$ 341.40 

$ 532.00 

55.8% 




Heliophysics 

STEREO 

$150.00 

$ 550.00 

266.7% 

49 

77 

57.1% 

ASO 

SWAS 

$ 47.30 

$ 78.90 

66.8% 

57 

116 

104.2% 


SWASTR 

$140.00 

$212.70 

51 .9% 




ASO 

SWIFT 

$ 102.40 

$ 164.90 

61 .0% 

41 

60 

45.1% 


TDRS7 

$ 269.00 

$ 532.00 

97.8% 





TDRS-H 




41 

52 

26.8% 


TDRS-I 




47 

73 

55.3% 


TDRS-J 




53 

82 

54.7% 


TE 

$ 321.30 

$ 401.50 

25.0% 




Earth Sci 

TERRA 

$1,078.70 

$1,393.20 

29.2% 




Earth Sci 

Terriers 




31 

56 

80.6% 


Tether 

$ 28.30 

$ 115.70 

308.8% 




Heliophysics 

THEMIS 

$ 102.30 

$ 107.60 

5.2% 

37 

44 

17.6% 

Heliophysics 

TIMED 

$ 129.20 

$ 176.20 

36.4% 

44 

86 

97.1% 


TOPEX 

$ 438.00 

$ 520.00 

18.7% 




Earth Sci 

Topex/Poseidon 




105 

151 

43.8% 

Heliophysics 

TRACE 

$ 35.60 

$ 40.30 

13.2% 

26 

52 

101.2% 


TRDS 

$ 899.80 

$ 803.10 

-10.7% 





Triana Spacecraft 

$ 75.00 

$ 96.90 

29.2% 




Earth Sci 

TRMM 

$ 218.80 

$ 468.00 

113.9% 

72 

87 

21 .0% 


TSS 

$ 40.70 

$ 263.00 

546.2% 




Earth Sci 

UARS 

$ 575.30 

$ 790.00 

37.3% 

73 

95 

30.1% 

Heliophysics 

Ulysses 

$ 196.00 

$ 460.00 

1 34.7% 

40 

132 

230.0% 

Heliophysics 

WIND 




48 

71 

47.9% 

ASO 

WIRE 

$ 39.70 

$ 50.70 

27.7% 

44 

57 

30.7% 

ASO 

WISE 




42 

59 

40.5% 

Manned 

X-30 

$3,100.00 

$ 10,000.00 

222.6% 

Canceled 



Manned 

X-33 - Canceled 

$1,075.20 

$1,789.70 

66.5% 

Canceled 



Manned 

X-34 - Canceled 

$ 70.00 

$ 378.00 

440.0% 

Canceled 



Manned 

X-38 - Canceled 

$ 500.00 

$ 1,500.00 

200.0% 

Canceled 



Manned 

X-43 Hyper-X Canceled 

$ 167.00 

$227.00 

35.9% 

Canceled 



ASO 

XTE or RXTE 

$ 100.00 

$ 373.00 

273.0% 

53 

48 

-9.4% 
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